"Device or resource busy." "No space left on disk" — while df disagrees. Both mysteries have one detective: lsof, the lister of open files, which maps every open file, socket, and deleted-but-held resource to the process holding it.

Who has this file open? #

Processes using a specific file
$ lsof /var/log/nginx/access.log
COMMAND  PID  USER   FD   TYPE  NODE NAME
nginx    812  root    6w  REG  998  /var/log/nginx/access.log
nginx    813  www     6w  REG  998  /var/log/nginx/access.log

The FD column shows the file descriptor — 6w is descriptor 6, opened for writing. One file, two processes, no guessing.

The famous cases #

Deleted files that still hold space — the disk is "full" but du finds nothing:

Find deleted-but-open files
$ lsof +L1
COMMAND PID  USER  FD  TYPE SIZE/OFF  NODE NAME
java    991  svc    4w  REG   19G     9982 /var/log/app.log (deleted)

The fix is not re-deleting the file; it is restarting or re-opening it in the holding process so the kernel finally frees the inode.

Which process owns a port — the missing piece when a service refuses to start:

Who is listening on 8080?
$ lsof -i :8080
COMMAND  PID  USER  FD  TYPE DEVICE NODE NAME
node    2201  ana  18u  IPv4  31001  TCP *:8080 (LISTEN)

Everything one process has — useful before you kill it:

Everything a process holds open
$ lsof -p 2201
QueryCommand
Who uses this filelsof file
Who listens on a portlsof -i :PORT
Deleted, held spacelsof +L1
All of a processlsof -p PID
Network everythinglsof -i

Sockets at scale belong to ss; the space mysteries start at df and du. More in the terminal notes series.