Linux File Size vs Disk Usage: Inspect Sparse Files with Python

ls -l says a file is 8 MiB, while du reports only a few KiB. Is the file damaged, or is disk accounting wrong? For a sparse file containing holes, both measurements can be correct: they measure different things.

This beginner-to-intermediate tutorial creates a small sparse file and uses Python to report logical bytes, allocated bytes, and hard-link identity. The checker only reads metadata. It does not open file contents, delete files, or change system settings.

1. Method: compare the same measurement

1. ls -l shows a regular file’s logical length. GNU du -B1 reports allocation in byte units, while du --apparent-size -B1 reports apparent size. Watch out: du -b is equivalent to the latter, not “allocated bytes.” These options are defined in the GNU du manual.
2. Python’s st_size is logical bytes; on Linux, st_blocks * 512 is the file’s reported allocated bytes. st_blksize is a suggested I/O size, not the multiplier for st_blocks. See the Python stat result documentation.
3. Two paths with the same (st_dev, st_ino) can be hard links to one file. Keep the identity rather than adding each path’s allocation. Linux documents links and metadata in stat.

Allocation below logical size is a clue, not an estimate of reclaimable space. Compression, shared blocks, preallocation, and filesystem behavior complicate interpretation. This tool neither locates holes nor decides whether a file is disposable. For a mismatch between df and directory totals, continue with disk space and deleted-file troubleshooting.

2. Setup: use a separate practice directory

Use Linux, an existing Python 3.10+ installation, and GNU coreutils. No pip packages are needed. These examples were tested with Python 3.11.4 and GNU coreutils 9.7; the shell commands work in bash or zsh.

1. Check the tools and create a new directory. Choose another name when repeating the exercise so earlier files remain untouched:

python3 --version
du --version
mkdir sparse-usage-demo
cd sparse-usage-demo

2. Run the following inside that directory. It writes four bytes at each end without writing every intervening logical byte. The xb mode refuses existing files:

python3 - <<'PYCODE'
import os

size = 8 * 1024 * 1024
with open("sparse.bin", "xb") as stream:
    stream.write(b"HEAD")
    stream.seek(size - 4)
    stream.write(b"TAIL")
    stream.flush()
    os.fsync(stream.fileno())
with open("empty.bin", "xb"):
    pass
print("created sparse.bin and empty.bin")
PYCODE

On filesystems supporting sparse files, writing beyond the current end can leave holes. Holes read as zero bytes, as described in Linux lseek. Actual allocation depends on the filesystem; not every mount type provides the same savings.

Do not run truncate on an existing database, virtual disk, or log to “fix its size.” Shortening a file discards data. This tutorial performs no such operation.

3. Implementation: a read-only file usage checker

Save this as file_usage.py:

"""Read Linux regular-file size metadata. Python 3.10+."""
import argparse
import json
import os
import stat
import sys


def inspect_file(path):
    info = os.lstat(path)
    if not stat.S_ISREG(info.st_mode):
        raise ValueError(f"{path!r}: not a regular file; no final symlink following")
    blocks = getattr(info, "st_blocks", None)
    if blocks is None:
        raise ValueError("st_blocks is unavailable on this platform")
    allocated = blocks * 512
    return {
        "path": path,
        "logical_bytes": info.st_size,
        "allocated_bytes": allocated,
        "allocated_less_than_logical": allocated < info.st_size,
        "hard_links": info.st_nlink,
        "device": info.st_dev,
        "inode": info.st_ino,
    }


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("paths", nargs="+", help="regular-file paths; no recursion")
    args = parser.parse_args()
    try:
        rows = [inspect_file(path) for path in args.paths]
    except (OSError, ValueError) as exc:
        print(f"ERROR: {exc}", file=sys.stderr)
        return 2
    print(json.dumps(rows, indent=2, ensure_ascii=True))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

os.lstat() inspects the final path entry itself. stat.S_ISREG() limits inputs to regular files, rejecting directories, FIFOs, and final-component symlinks. Links in parent directories still resolve normally; this is not a containment tool for untrusted directories.

Success returns 0 with a JSON array on stdout containing logical_bytes, allocated_bytes, allocated_less_than_logical, hard_links, device, and inode. Any failed path returns 2, writes an explanation to stderr, and emits no partial JSON. Argument errors also return 2. The script does not recurse or provide a simultaneous snapshot of several files.

4. Usage: compare JSON with du

Run these commands in order:

python3 file_usage.py --help
python3 file_usage.py sparse.bin empty.bin
ls -l -- sparse.bin
du -B1 -- sparse.bin
du --apparent-size -B1 -- sparse.bin
du -b -- sparse.bin

In this test environment, sparse.bin had a logical length of 8388608 bytes, an allocation of 8192 bytes, and a true Boolean field. du -B1 reported 8192; the other two du commands reported 8388608. These are observed values, not universal allocation assertions. The empty file’s Boolean is false, without a division-by-zero problem.

For real checks, pass regular-file paths you recognize and own. Put -- before arguments whose paths start with -. To find candidates first, use the logical-size ranking tool, then inspect allocation per file instead of summing its rankings.

5. Validation: cover hard links and failing inputs

Save the following as check_usage.py next to file_usage.py. It creates its own temporary subdirectory under the current directory and cleans it up automatically. It does not change the earlier demo files.

"""Run next to file_usage.py; all fixtures live in a temporary directory."""
import json
import os
from pathlib import Path
import subprocess
import sys
import tempfile

script = Path("file_usage.py").resolve()


def run(paths, expected):
    result = subprocess.run(
        [sys.executable, str(script), *map(str, paths)],
        text=True, capture_output=True, check=False,
    )
    assert result.returncode == expected, result.stderr
    if expected == 0:
        assert result.stderr == "", result.stderr
        return json.loads(result.stdout)
    assert result.stdout == "", result.stdout
    assert result.stderr.startswith("ERROR:"), result.stderr


with tempfile.TemporaryDirectory(prefix="usage-check-", dir=".") as tmp:
    directory = Path(tmp)
    sparse = directory / "sparse.bin"
    size = 8 * 1024 * 1024
    with sparse.open("xb") as stream:
        stream.write(b"HEAD")
        stream.seek(size - 4)
        stream.write(b"TAIL")
        stream.flush()
        os.fsync(stream.fileno())
    empty = directory / "empty.bin"
    empty.touch(exist_ok=False)
    tiny = directory / "tiny.bin"
    tiny.write_bytes(b"x")
    before = sparse.stat()
    rows = run([sparse, empty, tiny], 0)
    assert [row["logical_bytes"] for row in rows] == [size, 0, 1]
    for path, row in zip([sparse, empty, tiny], rows):
        info = path.lstat()
        assert row["allocated_bytes"] == info.st_blocks * 512
        assert row["allocated_less_than_logical"] == (
            row["allocated_bytes"] < row["logical_bytes"]
        )
    assert sparse.stat().st_mtime_ns == before.st_mtime_ns
    with sparse.open("rb") as stream:
        assert stream.read(4) == b"HEAD"
        stream.seek(size // 2)
        assert stream.read(4) == b"\0" * 4
        stream.seek(size - 4)
        assert stream.read() == b"TAIL"
    alias = directory / "alias.bin"
    os.link(sparse, alias)
    linked = run([sparse, alias], 0)
    assert linked[0]["hard_links"] == linked[1]["hard_links"] == 2
    assert (linked[0]["device"], linked[0]["inode"]) == (
        linked[1]["device"], linked[1]["inode"]
    )
    symlink = directory / "link.bin"
    symlink.symlink_to("sparse.bin")
    run([symlink], 2)
    run([directory], 2)
    run([directory / "missing.bin"], 2)
    run([sparse, directory / "missing.bin"], 2)
print("PASS: sizes, blocks, content, hard links, and failure exits")

Run:

python3 check_usage.py

A successful run prints:

PASS: sizes, blocks, content, hard links, and failure exits

The assertions check measurement formulas, content, and exit behavior, not a filesystem-specific amount of saved space. Tests also cover final-component symlinks, directories, missing files, and no partial stdout for mixed valid/invalid input. Matching hard-link identities mean aliases, not independent backups. To find separate content copies, use the chunked SHA-256 duplicate-file checker.

6. Troubleshooting and limits

A successful metadata query does not prove that contents are readable, a backup is complete, or the file will remain unchanged. A single stat result cannot precisely budget a live database backup. This tool also omits directory allocation, snapshots, and global filesystem metadata.

7. Summary

Separate logical length from reported allocation before explaining a disagreement between ls and du. Validate the formulas and exit codes with a small sparse file and hard links, then use the read-only JSON report in routine checks. Treat the result as a troubleshooting clue, not deletion advice or a promise of reclaimable space.