Fix Linux Argument list too long with find and Python

As a text-file directory grows, a working wc -c data/*.txt command can suddenly fail with Argument list too long. Quoting a path preserves spaces, but does not reduce an expanded argument list.

This tutorial gives beginner-to-intermediate Linux/Python users two read-only ways to total .txt file bytes: GNU tools receive filenames through standard input, or Python receives one directory and traverses it internally. Both handle spaces and newlines in filenames.

1. Choose a method: move the list out of argv

The Python errno documentation identifies E2BIG as an oversized argument list. A common cause is shell wildcard expansion before program startup. Arguments and the environment consume limited startup space, and an individual argument can also be too long. A machine’s ARG_MAX is not a fixed file-count limit.

Even subprocess.run(["wc", "-c", *paths]) puts every path into one process launch and can exceed the limit without using a shell. For timeouts and command failures, see the external-command tutorial.

2. Set up the environment and sample files

Use Linux, Bash, and Python 3.9 or newer. Method 1 also needs GNU findutils and GNU coreutils. The examples were verified with Python 3.11.4 and GNU wc 9.7, without third-party Python packages. If GNU tools are unavailable, use Method 2 without installing software globally.

1. Create a fresh demo directory; choose another name when repeating:

python3 --version
find --version
wc --version
mkdir argument-list-demo
cd argument-list-demo
python3 - <<'PY_DATA'
from pathlib import Path
root = Path("data")
root.mkdir()
for name, content in [("normal.txt", b"abc\n"),
                      ("two words.txt", b"12\n"),
                      ("line\nbreak.txt", b"xy\n")]:
    (root / name).write_bytes(content)
(root / "ignore.csv").write_text("not counted\n", encoding="utf-8")
PY_DATA

The three text files contain 10 bytes in total. One name contains a space and another contains an actual newline. The CSV is excluded. Run all remaining commands from this demo directory.

3. Method 1: separate filenames with NUL

1. Run in Bash:

set -o pipefail
find ./data -type f -name '*.txt' -print0 | wc -c --files0-from=-

The last line should show 10 total, with spacing that may vary. Quote *.txt so find handles the pattern. -print0 separates paths with NUL, preserving spaces and newlines. GNU wc reads those paths from standard input with --files0-from=-; its command line has a fixed number of arguments.

See GNU find’s filename-handling guide and the GNU wc manual. With pipefail, a find failure also fails the pipeline. Discard any partial statistics if the exit status is nonzero.

Displayed filenames may span lines, so do not parse this output by whitespace or newline. An empty match set provides no reliable total line, and a single file may have no total line either. For machine consumption, use the fixed JSON structure below.

4. Method 2: traverse inside Python

1. Save the complete code as size_tree.py, or download size_tree.py:

"""Summarize regular *.txt files without expanding filenames into argv."""
import argparse
import errno
import json
import os
from pathlib import Path
import stat
import sys


def stop_walk(error):
    raise error


def summarize(directory):
    root = Path(directory).resolve(strict=True)
    if not root.is_dir():
        raise NotADirectoryError(errno.ENOTDIR, "not a directory", str(root))
    files = total = 0
    for parent, _, names in os.walk(root, followlinks=False, onerror=stop_walk):
        for name in names:
            if not name.endswith(".txt"):
                continue
            info = (Path(parent) / name).lstat()
            if stat.S_ISREG(info.st_mode):
                files += 1
                total += info.st_size
    return {"files": files, "bytes": total}


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("directory", type=Path)
    args = parser.parse_args()
    try:
        report = summarize(args.directory)
    except OSError as exc:
        label = errno.errorcode.get(exc.errno, "OSERROR")
        print(f"ERROR {label}: {exc}", file=sys.stderr)
        return 1
    print(json.dumps(report, sort_keys=True))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

2. Check help and run the summary:

python3 size_tree.py --help
python3 size_tree.py ./data

Expected output:

{"bytes": 10, "files": 3}

The script receives one directory argument, then walks it internally. It does not build a list of the entire tree, although os.walk builds name lists for each directory, so a very wide directory still needs memory.

The official os.walk documentation explains that scan errors are ignored by default. Here onerror raises them so an incomplete scan fails. lstat() and S_ISREG count only regular files, skip file symlinks, and avoid descending into directory symlinks. The supplied root is resolved first; a root symlink therefore selects its target.

The total is logical bytes from file metadata. It neither reads contents nor tests readability, and it does not measure allocated disk space. GNU wc needs readable files, so permission restrictions can make the two methods behave differently. Hard links are counted per path. For backup capacity planning, continue with the disk-space preflight.

5. Validate unusual names and failure exits

Run this self-check in the same directory. It creates and automatically removes a separate temporary directory, leaving the demo data intact:

python3 - <<'PY_CHECK'
from pathlib import Path
from tempfile import TemporaryDirectory
import json
import subprocess
import sys
from size_tree import summarize

script = Path("size_tree.py").resolve()
with TemporaryDirectory() as work:
    root = Path(work)
    assert summarize(root) == {"files": 0, "bytes": 0}
    nested = root / "nested"
    nested.mkdir()
    for name in ("two words.txt", "line\nbreak.txt", "-option.txt"):
        (nested / name).write_bytes(b"abc\n")
    (root / "alias.txt").symlink_to(nested / "two words.txt")
    (root / "loop").symlink_to(root, target_is_directory=True)
    result = subprocess.run([sys.executable, str(script), str(root)],
                            capture_output=True, text=True, check=True)
    assert json.loads(result.stdout) == {"files": 3, "bytes": 12}
    for bad, label in [(root / "missing", "ENOENT"),
                       (nested / "-option.txt", "ENOTDIR")]:
        result = subprocess.run([sys.executable, str(script), str(bad)],
                                capture_output=True, text=True)
        assert result.returncode == 1 and not result.stdout, result
        assert label in result.stderr, result.stderr
print("PASS: empty, nested, unusual names, symlinks, error exits")
PY_CHECK

Expect PASS: empty, nested, unusual names, symlinks, error exits. Exit codes are 0 for success and 1 for path or scan errors, with no partial JSON on standard output. Argparse returns 2 for invalid command usage.

6. Troubleshooting and automation

Use absolute interpreter, script, and data-directory paths in scheduled jobs, and check the exit status. See scheduled-job path troubleshooting. Do not invoke python3 size_tree.py ./data/*.txt: this script accepts one directory.

Neither method creates a snapshot. Files added, removed, replaced, or written during scanning can affect the result. Use a stable directory you control; followlinks=False does not protect against an untrusted user replacing paths during traversal. Do not turn this sample into a privileged recursive deletion tool.

7. Summary

Move large filename lists into standard input or inside Python to avoid expanding every name into one command line. Preserve special names with NUL separators, and use JSON plus explicit exit codes for automation. Verify the counting rules and failure behavior on a small directory before scanning real data.