Check Disk Space with Python Before Starting a Batch Job

A batch job reports No space left on device after spending a long time computing results. Check the free space on the filesystem that contains the actual output directory before starting, and return a failure code when the budget does not fit.

This beginner-to-intermediate tutorial uses the Python standard library and tests failure paths without filling a disk. Use it before CSV exports, backups, or Hugo builds. For an existing write failure, start with Linux disk-space and inode troubleshooting.

1. Method: peak additional space plus a reserve

The rule is free_bytes >= (need_mib + reserve_mib) × 1048576. --need-mib estimates the peak additional data present at the same time during the task. --reserve-mib is the headroom you want to leave afterward, defaulting to 1024 MiB. One MiB is 1024 × 1024 bytes.

An export that adds 256 MiB while leaving 1024 MiB free needs at least 1280 MiB available. If old output, replacement output, and intermediate files coexist, include each additional copy in the budget. A compressed archive’s size is not its extracted-space requirement. These numbers are examples; adjust them for your task.

The official shutil.disk_usage() documentation defines total, used, and free space for a supplied path in bytes. Our script requires an existing output directory and resolves symlinks. Checking only / can measure the wrong filesystem when output lives on another mount.

2. Setup

Use Linux, Python 3.9 or newer, and Bash for the later entry-script example. No third-party Python packages are needed. The examples were tested on Linux with Python 3.11.4.

1. Create a fresh demo directory. Choose another name when repeating the exercise to avoid overwriting previous files:

python3 --version
mkdir disk-preflight-demo
cd disk-preflight-demo
mkdir output

2. Save the next section as disk_preflight.py in the current directory, or download the complete script. Run all subsequent commands from this demo directory.

3. Complete implementation

"""Check an existing output directory's free-space budget without writing files."""
import argparse
import json
from pathlib import Path
import shutil
import sys

MIB = 1024 * 1024


def nonnegative_int(text):
    value = int(text)
    if value < 0:
        raise argparse.ArgumentTypeError("must be a nonnegative integer")
    return value


def budget_report(directory, need_mib, reserve_mib):
    directory = Path(directory).resolve(strict=True)
    if not directory.is_dir():
        raise NotADirectoryError(f"not a directory: {directory}")
    free = shutil.disk_usage(directory).free
    required = (need_mib + reserve_mib) * MIB
    return {
        "path": str(directory),
        "free_bytes": free,
        "required_bytes": required,
        "shortfall_bytes": max(0, required - free),
        "ok": free >= required,
    }


def main(argv=None):
    parser = argparse.ArgumentParser(description=__doc__, allow_abbrev=False)
    parser.add_argument("directory", type=Path, help="existing output directory")
    parser.add_argument("--need-mib", type=nonnegative_int, required=True,
                        help="estimated peak additional space in MiB")
    parser.add_argument("--reserve-mib", type=nonnegative_int, default=1024,
                        help="space to leave free in MiB (default: 1024)")
    args = parser.parse_args(argv)
    try:
        report = budget_report(args.directory, args.need_mib, args.reserve_mib)
    except (OSError, RuntimeError) as exc:
        print(f"ERROR: {exc}", file=sys.stderr)
        return 2
    print(json.dumps(report, ensure_ascii=False))
    return 0 if report["ok"] else 1


if __name__ == "__main__":
    raise SystemExit(main())

argparse custom type conversion rejects negative values and non-integers; the parser also rejects misspelled options. Insufficient space is a completed check, so it still emits JSON on standard output. Missing directories, read failures, and invalid arguments return 2 without JSON.

The script only queries information. It creates no directories, allocates no space, and deletes no files. shortfall_bytes measures the gap against this particular budget.

4. Run and validate

1. Display help, then check the output directory:

python3 disk_preflight.py --help
python3 disk_preflight.py output --need-mib 256 --reserve-mib 1024

Output includes path, free_bytes, required_bytes, shortfall_bytes, and ok. Here, required_bytes is always 1342177280; available space and ok depend on your machine. Check that the reported path is the intended destination.

2. Simulate an exact fit and a one-MiB shortfall without consuming disk space:

python3 - <<'PY_CHECK'
from pathlib import Path
from types import SimpleNamespace
from unittest.mock import patch
from disk_preflight import MIB, budget_report

for free_mib, expected in [(1280, True), (1279, False)]:
    with patch("disk_preflight.shutil.disk_usage",
               return_value=SimpleNamespace(free=free_mib * MIB)):
        report = budget_report(Path("output"), 256, 1024)
    assert report["ok"] is expected
    assert report["shortfall_bytes"] == (0 if expected else MIB)
print("PASS: exact budget accepted; one MiB short rejected")
PY_CHECK

Expect PASS: exact budget accepted; one MiB short rejected. This tests logic with a mocked interface result; it does not test real quotas or disk writes.

3. Test the exit code against a real filesystem. Request a budget larger than its entire capacity; this is only a numerical comparison and writes no large files:

python3 - <<'PY_FAIL'
import json
import shutil
import subprocess
import sys

need = shutil.disk_usage("output").total // (1024 * 1024) + 1
result = subprocess.run(
    [sys.executable, "disk_preflight.py", "output",
     "--need-mib", str(need), "--reserve-mib", "0"],
    capture_output=True, text=True, check=False,
)
assert result.returncode == 1
assert json.loads(result.stdout)["ok"] is False
print("PASS: insufficient budget returns 1 without allocating disk space")
PY_FAIL
python3 disk_preflight.py missing-output --need-mib 1
python3 disk_preflight.py output --need-mib -1

The first block should print PASS: insufficient budget returns 1 without allocating disk space. The final two commands test a missing directory and a negative argument; each returns 2. Run echo $? immediately after each to inspect it. Additional checks covered regular-file paths, directory symlinks, and read failures.

5. Add a task entry script

Save this as run_demo.sh in the demo directory. It writes a small marker only after the check passes, making it easy to see whether the next command ran. The marker itself does not need 256 MiB; the budget demonstrates the interface.

#!/usr/bin/env bash
set -eu
cd -- "$(dirname -- "$0")"
mkdir -p output
if python3 disk_preflight.py output --need-mib 256 --reserve-mib 1024; then
  python3 -c 'from pathlib import Path; Path("output/completed.txt").write_text("done\n", encoding="utf-8")'
else
  result=$?
  printf 'Preflight stopped the task (exit %s)\n' "$result" >&2
  exit "$result"
fi

Run the entry script:

bash run_demo.sh

On success, inspect output/completed.txt. On failure, the entry script preserves the preflight exit code. A marker from a previous run may still exist; file existence alone does not prove the current run succeeded. For a real task, replace the marker command with your export or build command and use its actual output path and budget.

Use a predictable interpreter and working directory for scheduled runs; see Python scheduled-job path troubleshooting. If output, temporary files, and caches occupy different filesystems, check each directory with its own budget. Combine estimates for concurrent writes sharing one filesystem.

6. Troubleshooting and limits

Inspect the destination as the same task user, inside the same container or mount environment:

df -h -- output
df -i -- output

The official GNU df manual describes filesystem selection by path, readable sizes with -h, and inode statistics with -i. Creating files can fail when inodes are exhausted even if byte capacity remains.

Preflight does not reserve or lock disk space. Results can change immediately. This script does not validate write permissions or user quotas, and cannot replace handling real write errors. Your application must still handle OSError and avoid treating incomplete output as success. The reserve option is a comparison threshold; it changes no filesystem reserved-block settings.

7. Summary

Before starting a task, compare peak additional demand plus headroom against free space at its output directory, then use exit codes to stop work that exceeds the budget. Test success, insufficient space, and configuration failures before connecting the check to a scheduler; continue handling actual write failures during execution.