Python Timeout Leaves Workers Running? Stop a Linux Process Group

An external command times out, but its worker keeps writing files in the background. Giving the direct child a timeout does not automatically manage every process it starts.

This beginner-to-intermediate Linux tutorial gives one trusted command its own process group: send SIGTERM on timeout, allow a grace period, send SIGKILL, then reap the direct child. A worker that ignores SIGTERM makes the cleanup test observable, without changing schedulers or system configuration.

1. Method: manage a process group, not an arbitrary process tree

The existing subprocess timeout and logging guide handles the direct child. This follow-up covers the worker-process scenario it explicitly leaves out.

1. Start the command with Popen(..., start_new_session=True) so its group is separate from the current terminal’s group.
2. On timeout, send signals with os.killpg(process.pid, ...) to that newly started group. Do not match process names or invoke pkill.
3. After SIGTERM, wait a fixed grace period and send SIGKILL even if the group leader has exited. Otherwise, a worker ignoring SIGTERM could keep running.
4. Finally, wait() reaps the direct child. Use a fresh log and direct stdout/stderr to disk rather than PIPE.

Python’s Popen parameter reference explains start_new_session. The Linux process identifiers manual describes group inheritance. The boundary here is processes still in this group, not every descendant.

2. Setup: a fresh directory and an existing Python

Use Linux, bash or zsh, and Python 3.9 or newer; no third-party dependencies are needed. Examples were tested on Linux with Python 3.11.4. Run all subsequent commands in the same terminal and fresh directory. Use another directory when repeating the exercise so stale PID files cannot affect validation.

python3 --version
mkdir process-group-demo
cd process-group-demo

Save the next section as run_group.py. Run only trusted commands and arguments: shell=False is not a sandbox. See literal arguments and shell injection troubleshooting for argument-list usage.

3. Implementation: a new session, two-stage stop, and a fresh log

"""Run a trusted Linux command; stop its process group on timeout. Python 3.9+."""
import argparse
import math
import os
from pathlib import Path
import signal
import subprocess
import sys
import time


def signal_group(pgid, signum):
    try:
        os.killpg(pgid, signum)
    except ProcessLookupError:
        pass  # The group has already disappeared.


def run_group(argv, log_path, *, timeout=60.0, grace=1.0):
    if isinstance(argv, (str, bytes)) or not argv:
        raise ValueError("argv must be a nonempty argument list")
    if not math.isfinite(timeout) or timeout <= 0:
        raise ValueError("timeout must be finite and positive")
    if not math.isfinite(grace) or grace < 0:
        raise ValueError("grace must be finite and nonnegative")
    with Path(log_path).open("xb") as log:
        process = subprocess.Popen(
            argv, stdin=subprocess.DEVNULL,
            stdout=log, stderr=subprocess.STDOUT,
            shell=False, start_new_session=True,
        )
        try:
            code = process.wait(timeout=timeout)
        except subprocess.TimeoutExpired:
            # Do not poll/wait/reap the leader before the final group signal.
            try:
                signal_group(process.pid, signal.SIGTERM)
                time.sleep(grace)
            finally:
                signal_group(process.pid, signal.SIGKILL)
                process.wait()  # Reap our direct child after group cleanup.
            raise TimeoutError("process group timeout")
        return code if code >= 0 else 128 - code


def main():
    parser = argparse.ArgumentParser(description=__doc__, allow_abbrev=False)
    parser.add_argument("--timeout", type=float, default=60.0)
    parser.add_argument("--grace", type=float, default=1.0)
    parser.add_argument("log", type=Path)
    parser.add_argument("command", nargs=argparse.REMAINDER)
    args = parser.parse_args()
    command = args.command
    if command[:1] == ["--"]:
        command = command[1:]
    if not command:
        parser.error("provide a command after the log path and --")
    try:
        code = run_group(command, args.log, timeout=args.timeout, grace=args.grace)
    except TimeoutError:
        print(f"TIMEOUT: log={args.log}", file=sys.stderr)
        return 124
    except (OSError, ValueError) as exc:
        print(f"ERROR: {exc}", file=sys.stderr)
        return 2
    if code:
        print(f"FAILED: exit={code}; log={args.log}", file=sys.stderr)
    else:
        print(f"OK: log={args.log}")
    return code


if __name__ == "__main__":
    raise SystemExit(main())

Popen.wait(timeout=...) raises on timeout; it does not perform this group cleanup for you. The helper suppresses only ProcessLookupError for a vanished group. Other operating-system errors are reported on stderr.

The timeout branch deliberately does not call poll() or wait() before the final group signal. In this standalone, single-threaded program without a custom SIGCHLD handler, retaining the unreaped direct child avoids releasing its PID before signaling the group again. Do not transplant this logic into a host that has another child-process reaper.

os.killpg() addresses the group. The Linux signal manual explains that SIGKILL cannot be caught or ignored. Workers needing graceful shutdown should handle SIGTERM first, as in the record-boundary shutdown guide. Forced termination does not guarantee valid output.

4. Usage: success and a real failure

1. Read help and run a successful command:

python3 run_group.py --help
python3 run_group.py --timeout 5 ok.log -- python3 -c 'print("processed 3 records")'
cat ok.log

Output from the last two commands:

OK: log=ok.log
processed 3 records

2. Verify that a nonzero exit is not turned into success. This intentionally failing demonstration should not be pasted as a sequence into a script with set -e enabled:

python3 run_group.py --timeout 5 fail.log -- python3 -c 'import sys; print("bad input", file=sys.stderr); sys.exit(7)'
printf 'exit=%s\n' "$?"
cat fail.log

Corresponding output:

FAILED: exit=7; log=fail.log
exit=7
bad input

Exit conventions: 0 for success, 2 for argument/log/launch errors, 124 when the wrapper detects a timeout, and the command’s nonnegative status otherwise. Signal N maps to 128 + N. Commands can also return 2 or 124 themselves, so consult the ERROR, FAILED, or TIMEOUT prefix rather than attributing the source from the number alone.

5. Validation: stop a worker that ignores SIGTERM

1. Save this complete fixture as tree_job.py. It starts a worker that ignores SIGTERM and records its PID after installing that behavior. Unless stopped, the worker writes late.txt after six seconds.

"""A direct child plus a worker that ignores SIGTERM; Linux demo only."""
from pathlib import Path
import subprocess
import sys
import time

worker_code = """
from pathlib import Path
import os
import signal
import time
signal.signal(signal.SIGTERM, signal.SIG_IGN)
Path('worker.pid').write_text(str(os.getpid()), encoding='ascii')
time.sleep(6)
Path('late.txt').write_text('worker survived', encoding='utf-8')
"""
worker = subprocess.Popen([sys.executable, "-c", worker_code])
# Print only after the worker installed its handler and recorded its PID.
deadline = time.monotonic() + 3
while not Path("worker.pid").exists():
    if time.monotonic() > deadline:
        raise RuntimeError("worker did not become ready")
    time.sleep(0.01)
print(f"READY: worker={worker.pid}", flush=True)
worker.wait()

2. Run it in the clean demo directory. This command also deliberately returns a nonzero status:

python3 run_group.py --timeout 1 --grace 0.5 tree.log -- python3 tree_job.py
printf 'exit=%s\n' "$?"
cat tree.log

Expect TIMEOUT: log=tree.log, exit=124, and a log containing READY: worker=...; the PID varies. If READY is absent, investigate worker startup or a busy machine before treating this experiment as cleanup verification.

3. Check that the PID no longer executes, then wait beyond the worker’s original output deadline:

python3 - <<'PY_CHECK'
from pathlib import Path
import time

pid = int(Path("worker.pid").read_text(encoding="ascii"))
def still_running(pid):
    try:
        status = Path(f"/proc/{pid}/status").read_text()
    except FileNotFoundError:
        return False
    state = next(line for line in status.splitlines() if line.startswith("State:"))
    return state.split()[1] not in {"Z", "X"}

deadline = time.monotonic() + 2
while still_running(pid) and time.monotonic() < deadline:
    time.sleep(0.05)
assert not still_running(pid), "worker is still running"
time.sleep(6)
assert not Path("late.txt").exists(), "worker wrote after timeout"
print("PASS: worker stopped; no late output")
PY_CHECK

Expect PASS: worker stopped; no late output. Linux /proc state Z means a zombie that no longer executes code, not a worker still writing files. Its parent or adopting process is responsible for reaping it. This checks stopped execution and no late output, not that every group PID has disappeared from the process table.

Tests also compared the same tree_job.py input: using only subprocess.run(timeout=1) left the worker running and producing late.txt. With this wrapper, the worker terminated from SIGKILL and produced no such file. The test process adopted and reaped the orphaned demonstration workers.

4. Finally, confirm an existing log survives:

python3 - <<'PY_CHECK'
from pathlib import Path
import subprocess
import sys

before = Path("ok.log").read_bytes()
result = subprocess.run(
    [sys.executable, "run_group.py", "ok.log", "--",
     sys.executable, "-c", "print('replacement')"],
    capture_output=True, text=True,
)
assert result.returncode == 2, result.stderr
assert Path("ok.log").read_bytes() == before
print("PASS: existing log preserved")
PY_CHECK

6. Troubleshooting and boundaries

This helper covers timeouts while the direct command has not exited. If the direct child exits normally but leaves background workers, the wrapper does not clean them up. Supervising the full job lifecycle, handling SIGTERM/SIGKILL of the wrapper itself, cross-group workers, or credential changes calls for a job supervisor with resource boundaries, not more process-name matching.

The timeout is not a hard real-time deadline: process creation, the fixed grace period, and uninterruptible kernel waits can add to total duration. Logs are unbounded and should not contain credentials. This example provides no retries, disk quotas, or file rollback. Use it for small Linux jobs you control whose process-group behavior is understood.

7. Summary

Put one external command in its own process group, then send TERM followed by KILL on timeout to cover workers still in that group. Verify worker state and delayed artifacts, not just the leader’s exit. Respect the cross-group, normal-exit, and forced-termination boundaries before integrating the wrapper into batch automation.