Fix Python Process Pool Startup Errors: spawn, the Main Guard, and Worker Tests

A serial script works, but replacing its loop with a process pool produces a Linux RuntimeError about starting a process during bootstrapping. Moving the worker into a lambda then produces a pickle error. The calculation may be fine: the problem is how child processes import the script.

This beginner-to-intermediate tutorial builds a CPU batch entry point with explicit spawn: count primes in chunks, compare serial and pool results, and test argument errors and worker exceptions. It changes no system settings, needs no packages, and does not promise that small jobs become faster.

1. Method: separate calculation from the entry point

1. Define the worker at module level, with simple arguments and return values.
2. Enter argument parsing, pool creation, and reporting through main(), protected by if __name__ == "__main__":.
3. Pass mp.get_context("spawn") explicitly instead of relying on a platform default.
4. Print successful JSON in the parent only after consuming every result.

Python’s safe-import guidance explains the guard: spawn starts a fresh interpreter, and importing the main module in a child must not launch more processes. Do not put the worker definition inside the guard, either: importing the module would not define it.

The Python 3.14 ProcessPoolExecutor documentation records a change away from fork as the default start method. Fixing spawn here makes the import rules reproducible; it is not a recommendation to switch every application to spawn.

This example handles pure Python computation. For many file reads, start with bounded thread-pool batches. If you have not located the slow part yet, profile it with cProfile before adding processes.

2. Setup: run a real script file

Use an existing Python 3.9 or newer installation, with no third-party packages. Commands were tested on Linux with CPython 3.11.4 and work in bash or zsh; Windows and other Python versions were not tested.

1. Create a fresh practice directory in one terminal. Run subsequent commands there:

python3 --version
mkdir processpool-demo
cd processpool-demo

2. Save the complete implementation below as count_primes.py, or download count_primes.py. Do not paste the pool-creating script into a REPL, Notebook, or python3 - <<'PY': this example requires child processes to import a module from disk.

3. Implementation: explicit spawn, chunks, and parent aggregation

"""Count primes with an import-safe process pool. Python 3.9+, standard library."""
import argparse
from concurrent.futures import ProcessPoolExecutor
import json
from math import isqrt
import multiprocessing as mp
import sys


def count_chunk(bounds):
    start, stop = bounds
    if start < 2 or stop < start:
        raise ValueError("invalid interval")
    count = 0
    for number in range(start, stop):
        if number == 2 or (number % 2 and all(
            number % divisor for divisor in range(3, isqrt(number) + 1, 2)
        )):
            count += 1
    return count


def calculate(chunks, workers, serial=False):
    if serial:
        return sum(map(count_chunk, chunks))
    with ProcessPoolExecutor(
        max_workers=workers, mp_context=mp.get_context("spawn")
    ) as pool:
        return sum(pool.map(count_chunk, chunks))


def main():
    parser = argparse.ArgumentParser(description=__doc__, allow_abbrev=False)
    parser.add_argument("--limit", type=int, default=10000)
    parser.add_argument("--chunk-size", type=int, default=2500)
    parser.add_argument("--workers", type=int, default=2)
    parser.add_argument("--serial", action="store_true", help="skip the pool")
    args = parser.parse_args()
    if not 2 <= args.limit <= 1000000:
        parser.error("--limit must be 2..1000000")
    if not 1 <= args.chunk_size <= 1000000:
        parser.error("--chunk-size must be 1..1000000")
    if not 1 <= args.workers <= 8:
        parser.error("--workers must be 1..8")
    task_count = (args.limit - 1 + args.chunk_size - 1) // args.chunk_size
    if task_count > 1000:
        parser.error("too many chunks; increase --chunk-size (maximum 1000 chunks)")
    chunks = [
        (start, min(start + args.chunk_size, args.limit + 1))
        for start in range(2, args.limit + 1, args.chunk_size)
    ]
    try:
        total = calculate(chunks, args.workers, args.serial)
    except (OSError, RuntimeError, ValueError) as exc:
        print(f"ERROR: {type(exc).__name__}: {exc}", file=sys.stderr)
        return 1
    print(json.dumps({"limit": args.limit, "chunks": len(chunks),
                      "prime_count": total}))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

count_chunk() counts primes in the half-open interval [start, stop). The parent covers every integer from 2 through --limit, inclusive, without overlap; the default creates four tasks. Each worker returns only an integer, without writing files or printing success markers.

Here, --chunk-size is the number of integers covered by each calculation task, not the separate chunksize parameter of Executor.map(). The example accepts at most 1,000 tasks, eight workers, and a limit of one million to prevent accidental floods of submissions. These are tutorial policies, not Python platform limits.

sum(pool.map(...)) retrieves every result, so an exception in a worker is raised when its result is retrieved. The with block closes the pool but waits for tasks to finish: it is not a hard timeout or immediate-stop mechanism. See the official Executor documentation.

4. Usage: verify output before timing

1. Show help, then run serial and process-pool modes:

python3 count_primes.py --help
python3 count_primes.py --serial
python3 count_primes.py --workers 2

The last two commands should each print this line and exit with status 0:

{"limit": 10000, "chunks": 4, "prime_count": 1229}

--workers 1 still starts one child worker; only --serial avoids creating a pool. Invalid arguments make argparse return 2. Caught resource, runtime, or interval errors return 1, write to stderr, and produce no successful JSON. Other uncaught exceptions also fail with a nonzero status and traceback; do not treat them as success.

2. To evaluate performance, run each command a few times in bash with identical input:

time python3 count_primes.py --limit 200000 --chunk-size 25000 --serial
time python3 count_primes.py --limit 200000 --chunk-size 25000 --workers 2

Compare actual real time and confirm that the JSON matches. This prime algorithm is a verifiable CPU exercise, not a high-performance sieve. With the small default input, interpreter startup and interprocess communication often outweigh useful work. CPU quotas, task size, and other workloads affect results.

5. Validation: success, bad arguments, and a worker exception

Save the following as a verify_pool.py file, or download verify_pool.py. The verifier itself also needs an entry-point guard and must not run from stdin.

"""Run from a saved .py file, not stdin: spawn must import the main module."""
from concurrent.futures import ProcessPoolExecutor
import json
import multiprocessing as mp
import subprocess
import sys
from count_primes import count_chunk


def main():
    expected = {"limit": 10000, "chunks": 4, "prime_count": 1229}
    for extra in (["--serial"], [], ["--workers", "1"]):
        result = subprocess.run(
            [sys.executable, "count_primes.py", *extra],
            capture_output=True, text=True, timeout=30, check=False,
        )
        assert result.returncode == 0 and not result.stderr, result
        assert json.loads(result.stdout) == expected, result
    for extra in (["--workers", "0"], ["--limit", "1"],
                  ["--chunk-size", "0"], ["--chunk-size", "1"]):
        result = subprocess.run(
            [sys.executable, "count_primes.py", *extra],
            capture_output=True, text=True, timeout=30, check=False,
        )
        assert result.returncode == 2 and result.stdout == "", result
        assert "error:" in result.stderr, result
    with ProcessPoolExecutor(max_workers=1, mp_context=mp.get_context("spawn")) as pool:
        future = pool.submit(count_chunk, (8, 2))
        try:
            future.result(timeout=10)
        except ValueError as exc:
            assert str(exc) == "invalid interval", exc
        else:
            raise AssertionError("worker failure was lost")
    print("PASS: serial/process/one-worker agree; bad arguments rejected; worker error received")


if __name__ == "__main__":
    main()

Run:

python3 verify_pool.py

Expected output, with exit status 0:

PASS: serial/process/one-worker agree; bad arguments rejected; worker error received

The test submits the reversed interval (8, 2) and confirms that the parent receives ValueError from future.result() instead of silently losing a task. Here, timeout=10 limits waiting for the result, not the lifetime of a running task. Subcommand timeouts in the verifier are not complete process-tree management, either.

6. Troubleshooting: locate the failing code site

The official pickle rules explain that functions are serialized by module and name, not by sending their code with each task. This example reads no external pickle files; do not load unfamiliar pickle files to “fix” argument passing when adapting a real task.

If the job mainly invokes external commands rather than computing in Python, use subprocess calls with logs and timeouts. Do not wait inside a worker for new tasks from the same pool, or pass database connections or open files directly as worker arguments.

7. Summary

Make child imports safe before adding parallelism: module-level workers, a guarded entry point, and an explicit context solve different problems. Compare serial and pool results on identical input, and actually retrieve Future results to expose exceptions. Integrate the pool into automation only when output is correct and measured whole-command time justifies it.