"Device or resource busy." "No space left on disk" — while df disagrees. Both mysteries have one detective: lsof, the lister of open files, which maps every open file, socket, and deleted-but-held resource to the process holding it.
Who has this file open? #
$ lsof /var/log/nginx/access.log
COMMAND PID USER FD TYPE NODE NAME
nginx 812 root 6w REG 998 /var/log/nginx/access.log
nginx 813 www 6w REG 998 /var/log/nginx/access.log
The FD column shows the file descriptor — 6w is descriptor 6, opened for writing. One file, two processes, no guessing.
The famous cases #
Deleted files that still hold space — the disk is "full" but du finds nothing:
$ lsof +L1
COMMAND PID USER FD TYPE SIZE/OFF NODE NAME
java 991 svc 4w REG 19G 9982 /var/log/app.log (deleted)
The fix is not re-deleting the file; it is restarting or re-opening it in the holding process so the kernel finally frees the inode.
Which process owns a port — the missing piece when a service refuses to start:
$ lsof -i :8080
COMMAND PID USER FD TYPE DEVICE NODE NAME
node 2201 ana 18u IPv4 31001 TCP *:8080 (LISTEN)
Everything one process has — useful before you kill it:
$ lsof -p 2201
| Query | Command |
|---|---|
| Who uses this file | lsof file |
| Who listens on a port | lsof -i :PORT |
| Deleted, held space | lsof +L1 |
| All of a process | lsof -p PID |
| Network everything | lsof -i |
Sockets at scale belong to ss; the space mysteries start at df and du. More in the terminal notes series.
