Python Too Many Open Files: Find File Handle Leaks on Linux

A batch script runs for a while, then raises OSError: [Errno 24] Too many open files. There is free disk space and the input files are small, yet the next file cannot be opened. The issue often concerns descriptors held simultaneously by the process, rather than the total number of input files.

This beginner-to-intermediate tutorial reproduces a file handle leak on Linux, inspects its limits and open files from another process, then fixes it with with. The demo only reads a test file; lowering the soft limit affects only the demo Python process.

1. Method: inspect the process that fails

The official Python resource documentation describes RLIMIT_NOFILE as the descriptor limit for the current process. getrlimit() returns the soft and hard limits. The soft limit is the active restriction and cannot exceed the hard limit.

Descriptors also represent sockets and pipes. The Python errno documentation lists EMFILE for too many open files and ENFILE for a full system file table; they require different investigation scopes.

First distinguish these cases:

A single high count does not prove a leak. Sample repeatedly under comparable load and connect descriptor targets to their lifetimes in the code.

2. Prepare the environment and test file

Use Linux with accessible /proc and an existing Python 3.9 or newer interpreter. No third-party packages are needed; the examples were tested on Linux with Python 3.11.4. The resource module is unavailable on Windows.

1. Create a separate demo directory in bash. Choose another directory name when repeating the experiment:

python3 --version
python3 -c 'import resource; print("resource ready")'
mkdir fd-demo
cd fd-demo
printf 'alpha\nbeta\n' > sample.txt

2. Save the next section as fd_demo.py in the current directory, or download the complete demo script.
3. The demo lowers its own soft limit to 64. It keeps an existing smaller limit and refuses to run below 16. The hard limit, parent shell, and other processes stay unchanged. This is a setting for reproducing the failure; do not put it in a real task’s entry point.

3. Complete implementation: one script for both modes

"""Reproduce a file descriptor leak, then compare scoped file reads on Linux."""
import argparse
import os
from pathlib import Path
import resource
import sys
import time


def main():
    parser = argparse.ArgumentParser(description=__doc__, allow_abbrev=False)
    parser.add_argument("file", type=Path, help="existing input file, read only")
    parser.add_argument("--mode", choices=("leak", "fixed"), default="fixed")
    parser.add_argument("--iterations", type=int, default=200)
    parser.add_argument("--hold", type=int, default=0, help="inspection pause, 0-60 seconds")
    args = parser.parse_args()
    if not 1 <= args.iterations <= 10000:
        parser.error("iterations must be from 1 to 10000")
    if not 0 <= args.hold <= 60:
        parser.error("hold must be from 0 to 60")

    soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
    finite = [64] + [n for n in (soft, hard) if n != resource.RLIM_INFINITY]
    limit = min(finite)
    if limit < 16:
        parser.error("demo needs an existing descriptor limit of at least 16")
    try:
        resource.setrlimit(resource.RLIMIT_NOFILE, (limit, hard))
    except (OSError, ValueError) as exc:
        print(f"ERROR: cannot set demo limit: {exc}", file=sys.stderr, flush=True)
        return 1
    print(f"pid={os.getpid()} mode={args.mode} soft={limit} hard={hard}", flush=True)

    retained = []
    processed = status = 0
    try:
        try:
            for _ in range(args.iterations):
                if args.mode == "leak":
                    file = args.file.open("rb")
                    retained.append(file)
                    file.read(1)
                else:
                    with args.file.open("rb") as file:
                        file.read(1)
                processed += 1
        except OSError as exc:
            print(f"ERROR: errno={exc.errno}: {exc}", file=sys.stderr, flush=True)
            status = 1
        print(f"processed={processed} retained={len(retained)} status={status}", flush=True)
        if args.hold:
            print(f"READY: holding for {args.hold}s", flush=True)
            time.sleep(args.hold)
    finally:
        for file in retained:
            file.close()
    return status


if __name__ == "__main__":
    raise SystemExit(main())

The leak branch stores open file objects in a list without closing them after reading. The fixed branch reads inside with and closes each file on leaving the block. Both reopen the same file, so even a directory containing only one input file can produce this failure.

--hold pauses the process after completion or an error so another process can inspect it. The leak branch closes its files in finally after the pause, but the work loop already failed and its exit code remains 1. Cleanup at the end cannot replace timely release inside the loop.

4. Method 1: reproduce the error and inspect open files

1. Display help and run leak mode. Execute these commands in the same bash terminal; pid=$! captures the background task just started:

python3 fd_demo.py --help
python3 fd_demo.py sample.txt --mode leak --hold 30 &
pid=$!
sleep 1
cat "/proc/$pid/limits"
ls -l "/proc/$pid/fd"
python3 - "$pid" <<'PY_FD'
from pathlib import Path
import sys
folder = Path("/proc", sys.argv[1], "fd")
print("fd_count=", sum(1 for _ in folder.iterdir()), sep="")
PY_FD
if wait "$pid"; then
  printf 'exit=0\n'
else
  printf 'exit=%s\n' "$?"
fi

2. Inspect the soft limit in the Max open files row and the entries in /proc/$pid/fd repeatedly pointing to sample.txt. With only the three initial standard input, output, and error descriptors, typical output is:

ERROR: errno=24: [Errno 24] Too many open files: 'sample.txt'
processed=61 retained=61 status=1
READY: holding for 30s
fd_count=64
exit=1

The PID, hard limit, and successful iteration count depend on the launch environment. Extra inherited descriptors also consume capacity; do not use 61 as a business assertion.

The Linux kernel /proc documentation describes directories named after process IDs and fd containing their descriptors. The /proc/PID/limits manual documents the soft and hard resource limits listed there. ls -l displays descriptor link targets.

Inspect from outside because a process that exhausted its descriptors may fail to open even its own /proc/self/fd directory. Sampling is not an atomic snapshot; real programs can close files while you inspect them.

5. Method 2: close promptly and verify the fix

1. Run fixed mode under the same demo limit:

python3 fd_demo.py sample.txt --mode fixed
printf 'exit=%s\n' "$?"
python3 fd_demo.py sample.txt --mode fixed --iterations 1000

The first run should print processed=200 retained=0 status=0 and exit=0; the second should print processed=1000 retained=0 status=0. Completed iterations exceed the soft limit because each iteration releases its descriptor.

2. Verify that an exception also closes the file:

python3 - <<'PY_CLOSE'
file = None
try:
    with open("sample.txt", "rb") as file:
        file.read(1)
        raise RuntimeError("demo processing error")
except RuntimeError:
    pass
assert file is not None and file.closed
print("OK: closed after exception")
PY_CLOSE

The Python file I/O tutorial explains that with closes files after the block, including when an exception occurs. Place it around each file’s processing scope in a batch job, rather than opening every input before processing them all.

For CSV input, use the single-file reading structure from streaming large CSV files. External commands also require pipe and log handle management and waiting for children; see Python subprocess timeouts and log files.

6. Troubleshooting: limits, permissions, and automation

These read-only commands display the current bash soft and hard limits:

ulimit -Sn
ulimit -Hn

They report the current shell’s limits, which may differ from an already running service. Use /proc/ACTUAL_PID/limits for production investigation. Check execution paths and working directories with the Linux scheduled job environment tutorial.

If counts fall back under reasonable concurrency but the peak exceeds the current restriction, adjust the limit through the actual launcher and inspect the running process again. A continuously growing leak simply fails later under a higher limit. After fixing it, repeat the same workload and confirm counts return to a stable range; retained=0 only describes the demo list, rather than all application handles.

7. Summary

Inspect the failing process’s limits and descriptor targets, then distinguish a leak from normal concurrency needs. Scope file processing with with and verify resource release on both normal and exception paths. Repeat sampling in the actual launch environment to confirm stable counts before deciding whether capacity needs adjustment.