Python subprocess Wildcards Not Expanding: Arguments, Spaces, and Redirection

A command accepts *.txt in your terminal, but receives that exact string when called from subprocess.run(). Adding > or | to the argument list creates neither a report nor a pipeline. The usual mistake is treating shell syntax as program arguments.

This beginner-to-intermediate tutorial prints the arguments a child actually receives, then explicitly selects files, counts their lines one at a time, and redirects a report to disk. It keeps shell=False throughout, without constructing shell command strings. The examples target small UTF-8 files in trusted local directories, not a general secure execution framework.

1. Method: separate arguments from shell syntax

Python’s subprocess argument documentation distinguishes direct execution from shell execution. With an argument list here, each element is one argument: spaces need no extra quoting, and a shell does not interpret wildcards, variables, or redirection operators.

Use glob to select files, stdout to direct output, and sys.executable to launch the fixed Python worker. If the executable cannot be found, use the PATH and command preflight guide rather than hiding the problem behind shell=True.

2. Setup: create a small input directory

Use an existing Python 3.10 or newer installation; no third-party packages are needed. The commands were tested on Linux with Python 3.11.4 and assume bash/zsh. Work in a new directory; choose a different directory name when repeating the experiment to avoid overwriting old fixtures.

python3 --version
mkdir subprocess-argv-demo
cd subprocess-argv-demo
mkdir inputs
printf 'alpha\nbeta\n' > 'inputs/report one.txt'
printf 'gamma\n' > inputs/second.txt
printf 'hidden\n' > inputs/.private.txt

The third file, .private.txt, tests the hidden-file policy. Save the following three scripts in the current subprocess-argv-demo directory, not inside inputs.

3. Reproduce: inspect the child’s arguments

1. Save this as show_argv.py:

import json
import sys

print(json.dumps(sys.argv[1:], ensure_ascii=True))

2. Run the argument probe:

python3 - <<'PY_PROBE'
import subprocess
import sys

result = subprocess.run(
    [sys.executable, "show_argv.py", "*.txt", "report one.txt",
     "|", "$HOME", ">", "result.txt"],
    shell=False, check=True, capture_output=True,
    text=True, encoding="utf-8", timeout=5,
)
print(result.stdout, end="")
PY_PROBE

It prints the following and does not create result.txt:

["*.txt", "report one.txt", "|", "$HOME", ">", "result.txt"]

report one.txt stays one argument. The quotes in "report one.txt" delimit a Python string; they are not sent to the program. Do not add another layer of shell quote characters inside the string. The | is also an ordinary character here, not a connection between programs.

4. Implementation: select files, then invoke the worker

1. Save the single-file counter as count_text.py:

"""Count nonempty lines in one trusted UTF-8 text file. Python 3.10+."""
import argparse
import json
from pathlib import Path
import sys


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("path", type=Path)
    args = parser.parse_args()
    try:
        with args.path.open(encoding="utf-8") as handle:
            count = sum(1 for line in handle if line.strip())
    except (OSError, UnicodeError) as exc:
        print(f"ERROR: {exc}", file=sys.stderr)
        return 1
    print(json.dumps({"file": args.path.name, "nonempty_lines": count},
                     ensure_ascii=True))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

2. Save the batch entry point as batch_count.py, beside the counter:

"""Count top-level non-hidden *.txt files without invoking a shell."""
import argparse
import glob
import json
from pathlib import Path
import subprocess
import sys


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("directory", type=Path)
    args = parser.parse_args()
    worker = Path(__file__).resolve().with_name("count_text.py")
    try:
        root = args.directory.resolve(strict=True)
        if not root.is_dir():
            raise ValueError("input must be a directory")
        names = sorted(glob.glob("*.txt", root_dir=root))
        if not names:
            raise ValueError("no matching *.txt files")
        results = []
        for name in names:
            path = root / name
            if not path.is_file():
                raise ValueError(f"not a regular file: {name!r}")
            result = subprocess.run(
                [sys.executable, str(worker), "--", str(path)],
                shell=False, stdin=subprocess.DEVNULL,
                capture_output=True, text=True, encoding="utf-8",
                check=True, timeout=10,
            )
            results.append(json.loads(result.stdout))
    except subprocess.CalledProcessError as exc:
        print(f"ERROR: worker exit={exc.returncode}; "
              f"{exc.stderr.strip()}", file=sys.stderr)
        return 1
    except (OSError, ValueError, subprocess.TimeoutExpired) as exc:
        print(f"ERROR: {exc}", file=sys.stderr)
        return 1
    print(json.dumps({"files": results}, ensure_ascii=True))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

The *.txt pattern is fixed in our code, not a user-supplied shell expression. The official glob documentation describes root_dir as the search root; sort results yourself. This pattern selects top-level non-hidden names, without recursion or ~ expansion. Consequently, .private.txt is excluded.

Each invocation receives one absolute file path instead of a batch-sized argument list. The -- marker ends option parsing in this argparse worker; it is not a protocol every tool supports. A directory matching *.txt is explicitly rejected. Symlinks to regular files are accepted, so this is not a path-isolation tool.

3. Inspect usage and run the batch:

python3 batch_count.py --help
python3 batch_count.py inputs

The second command prints:

{"files": [{"file": "report one.txt", "nonempty_lines": 2}, {"file": "second.txt", "nonempty_lines": 1}]}

No matches returns 1 rather than treating an empty batch as success; argument errors return 2. Worker failures, read errors, and timeouts return 1 with a diagnostic on stderr. The parent emits JSON only after every file succeeds, avoiding a partial batch report on stdout.

5. Output files: use stdout, not a greater-than argument

Run this in the same directory:

python3 - <<'PY_REDIRECT'
from pathlib import Path
import subprocess
import sys

with Path("counts.json").open("xb") as output:
    subprocess.run(
        [sys.executable, "batch_count.py", "inputs"],
        shell=False, stdin=subprocess.DEVNULL,
        stdout=output, check=True, timeout=30,
    )
PY_REDIRECT

Mode xb requires a new counts.json, protecting existing reports; choose a fresh filename when repeating this step. Redirection opens the file before launch, so a failed child can leave an empty file. Existence alone is not success. To replace reports while retaining the previous valid version on failure, see atomic JSON writes in Python.

Here stderr stays attached to the terminal; only stdout goes into the JSON file. Downstream steps must check the exit status instead of parsing a leftover report after failure.

6. Validation: success, hidden files, and no matches

Save this as check_demo.py and run python3 check_demo.py. It expects the unchanged inputs from section 2 and the absence of result.txt demonstrated in section 3.

import json
from pathlib import Path
import subprocess
import sys

probe = subprocess.run(
    [sys.executable, "show_argv.py", "*.txt", "report one.txt",
     "|", "$HOME", ">", "result.txt"],
    capture_output=True, text=True, encoding="utf-8", check=True, timeout=5,
)
assert json.loads(probe.stdout) == [
    "*.txt", "report one.txt", "|", "$HOME", ">", "result.txt",
]
result = subprocess.run(
    [sys.executable, "batch_count.py", "inputs"],
    capture_output=True, text=True, encoding="utf-8", timeout=30,
)
assert result.returncode == 0, result.stderr
assert json.loads(result.stdout) == {"files": [
    {"file": "report one.txt", "nonempty_lines": 2},
    {"file": "second.txt", "nonempty_lines": 1},
]}
assert not Path("result.txt").exists()
Path("empty").mkdir(exist_ok=True)
failed = subprocess.run(
    [sys.executable, "batch_count.py", "empty"],
    capture_output=True, text=True, encoding="utf-8", timeout=30,
)
assert failed.returncode == 1
assert failed.stdout == ""
assert "no matching *.txt files" in failed.stderr
assert Path("inputs/.private.txt").read_text() == "hidden\n"
print("PASS: exact arguments, counts, hidden-file exclusion, empty-match failure")

On success it prints PASS: exact arguments, counts, hidden-file exclusion, empty-match failure. The empty-match test deliberately fails without emitting success JSON. This check does not alter the inputs; it only creates an empty test directory.

7. Troubleshooting and boundaries

Captured output uses memory. Each worker emits only one small JSON object here, but individual input lines can still be large and the final report grows with file count. Files can change after selection: use a trusted directory with writers stopped, not this script as a snapshot or integrity audit. For further timeout and logging work, see subprocess timeouts, errors, and log files.

shell=False does not restrict program privileges or stop a tool from interpreting an argument as an option. This example only launches our own Python scripts. Do not replace it with untrusted string concatenation and an enabled shell; see the Python subprocess security notes.

8. Summary

Print the actual arguments before deciding which operations Python needs to perform. Handle selection, execution, and output redirection separately rather than mixing shell syntax into an argument list. Test success, empty matches, and failure exits before an automated workflow consumes the report.