Linux Disk Still Full After Deleting Files: Diagnose with df, du, and lsof

Deleting a large log does not always free its disk space. A server can also report No space left on device while df -h still shows free capacity. This guide separates three causes: allocated blocks, exhausted inodes, and deleted files held open by a process.

The commands target a Linux host with GNU coreutils. Use the path where writes fail, and work in the same host or container mount namespace as the failing application.

1. Install the inspection tools

On Debian or Ubuntu, install any missing tools:

sudo apt-get update
sudo apt-get install lsof util-linux coreutils python3

If the affected filesystem is already full, use the installed tools first; package installation needs space too. Other distributions provide these utilities through their own package manager. Python is only needed for the optional demo.

df --help
du --help
findmnt --help
lsof -h

2. Identify the affected filesystem

In the same shell, set target to an existing directory where writes fail. This example uses /var:

target=/var
findmnt -T "$target" -o TARGET,SOURCE,FSTYPE,OPTIONS
df -hT "$target"
df -i "$target"

findmnt -T resolves the filesystem containing that path. A separate /var, container overlay, or network mount needs its own investigation; checking / alone can miss the problem.

If an expected external drive is missing from the mount results, follow the Linux external drive detection and mounting guide to confirm the device and mount point before checking capacity.

Read the two df reports together:

df reports filesystem statistics; -i switches to inode usage. On filesystems with meaningful fixed inode limits, creating a new file can fail even when data blocks remain available. Check the actual Avail and IFree values as well as rounded percentages.

3. Find which directories consume space

Start at the mount point found above:

mountpoint=$(findmnt -n -o TARGET -T "$target")
sudo du -xhd1 "$mountpoint" | sort -h

GNU du walks reachable files. -x stays on one filesystem and -d1 limits the displayed directory depth. For inode pressure, use --inodes. Drill into a large directory, for example:

sudo du -xhd1 /var | sort -h
sudo du --inodes -x -d1 /var | sort -n

A low du total alongside high df usage is a clue, not proof of a deleted file. Permission errors, data hidden beneath mounts, filesystem metadata, and snapshots can also complicate the comparison. Keep scan errors visible. On a busy server, repeat measurements after the workload settles; these commands do not take a shared snapshot.

For ordinary growth, identify the owner of the largest directory and apply its retention or cleanup procedure. For many small files, adjust that application’s cache or spool retention. Avoid deleting database or container storage files by hand.

On an older Yum-based system, if you have traced the usage to old kernels in /boot or the Yum cache, continue with fixing insufficient space during Yum updates.

4. Find deleted files that are still open

sudo lsof -nP +L1

lsof +L1 selects open files with a link count below one. Look for regular files (TYPE = REG), their PID, FD, SIZE/OFF, and a name ending in (deleted). -nP disables address and port-name resolution.

A file can appear more than once when several processes or descriptors hold it. Do not sum every row as unique disk usage. SIZE/OFF is not an allocated-block measurement, especially for sparse files.

Deleting a name removes its directory entry. When the last link is removed but a process still has the file open, Linux keeps the file until the last open descriptor closes. This is the behavior documented by unlink(2).

Identify the owning service before taking action. For a confirmed deleted Nginx log, using the binary and configuration for that running instance, reopen its logs:

sudo nginx -t && sudo nginx -s reopen
sudo lsof -nP +L1
df -hT /var/log/nginx

Nginx documents log reopening as the way to release old log handles after rotation. With a custom configuration, supply the same -c and, if used, -p arguments as the running instance. Check its error log if reopening fails. Other services need their own documented reopen operation, or a planned restart of the owning service. A generic reload does not guarantee that file handles close.

Do not truncate an arbitrary /proc/PID/fd/FD as a shortcut: that descriptor may refer to application data. Once the immediate issue is resolved, fix log rotation so the application reopens its logs after rotation.

5. Reproduce the behavior with an 8 MiB demo

Run this on a Linux test machine with at least 8 MiB free in its temporary directory. It creates and unlinks only its own temporary file, inspects the current Python process, and closes the file automatically:

python3 - <<'PY'
import os
from pathlib import Path
import subprocess
import tempfile

with tempfile.TemporaryDirectory(prefix="deleted-file-demo-") as directory:
    path = Path(directory) / "demo.log"
    with path.open("w+b") as handle:
        for _ in range(8):
            handle.write(b"x" * 1024 * 1024)
        handle.flush()
        os.fsync(handle.fileno())
        path.unlink()
        info = os.fstat(handle.fileno())
        print(f"Path exists: {path.exists()}", flush=True)
        print(f"Link count: {info.st_nlink}", flush=True)
        print(f"Open file bytes: {info.st_size}", flush=True)
        print(f"Allocated bytes: {info.st_blocks * 512}", flush=True)
        subprocess.run([
            "lsof", "-nP", "-a", "-p", str(os.getpid()), "+L1"
        ], check=True)
        handle.seek(0)
        assert handle.read(4) == b"xxxx"
    print("Closed: the demo no longer holds the deleted file")
PY

The first lines are:

Path exists: False
Link count: 0
Open file bytes: 8388608

An allocated-byte count and an lsof row follow. Allocation depends on the filesystem; the pathname should end in (deleted). The assertion confirms the process can still read the unlinked file. The final line confirms the demo closed its handle.

This demo was tested with Python 3.11.4 on Linux. Its small allocation may not change the rounded df -h percentage; the descriptor and zero link count demonstrate the mechanism directly. Run it in a normal host shell if a sandbox blocks process inspection.

6. Verify recovery and handle remaining errors

Repeat the checks in the shell where target was set:

df -hT "$target"
df -i "$target"
sudo lsof -nP +L1

Confirm the relevant deleted-file entries disappear, available space or free inodes recover as appropriate, and the application can perform its original write. No matches from lsof +L1 can produce exit status 1; inspect stderr before treating that status as an inspection failure.

If capacity has recovered but the website still returns 5xx responses, continue with Nginx 5xx log analysis using rg and Python to narrow the incident by time, URL, and status code.

7. Summary

Check the filesystem containing the failing path, then distinguish block pressure from inode pressure. Use du to locate reachable data and lsof +L1 to find unlinked files still held open. Release the confirmed owner’s handles through its supported operation, then verify both filesystem capacity and the application’s write.