Python SIGTERM Shutdown: Finish the Current Record Before Stopping

A background batch job receives a stop signal while writing a result and exits with an unfinished record. This tutorial adds a stop flag to a single-process Python script: after SIGTERM or SIGINT, it finishes the item already in progress, closes the output file, and returns an explicit stopped status.

The demo writes IDs and squared values so you can verify it in your own working directory. It fits synchronous file processing and data conversion; add SQLite batch checkpoints when you also need to resume after a restart.

1. Method: check a stop flag between records

The processing order is:

check stop flag → process one item → write → flush → fsync → next check

The signal handler only remembers the first signal number. Writing, printing, and cleanup stay in the normal execution flow, avoiding a second I/O operation inside a handler that interrupted the first. The official Python signal documentation explains that handlers run in the main thread and must be registered there; long-running C extension work can delay their execution.

Boundary rule: an item already admitted to processing may finish. Once the loop observes the stop flag, it starts no further item. Shutdown latency depends on the current operation, so real jobs need timeouts for network requests and external commands; see the subprocess timeout and logging tutorial.

2. Setup: download and run a normal batch

Use Linux and Python 3.10 or newer. No third-party packages are required. The examples were tested on Linux with Python 3.11.4; the commands work in bash or zsh.

1. Download graceful_batch.py into a fresh working directory, or save the complete code in the next section under that filename.
2. Run:

python3 --version
python3 graceful_batch.py --help
mkdir graceful-demo
python3 graceful_batch.py graceful-demo/complete.jsonl --items 3 --delay 0
cat graceful-demo/complete.jsonl

The program prints DONE completed=3. The resulting file should contain:

{"id": 1, "square": 1}
{"id": 2, "square": 4}
{"id": 3, "square": 9}

Use a fresh output filename for each run. Mode "x" preserves existing files; this program does not append or resume automatically.

3. Implementation: the complete script

"""Finish the current JSONL record before stopping. Linux, Python 3.10+."""
import argparse
import json
import math
import os
from pathlib import Path
import signal
import sys
import time

stop_signal = None


def request_stop(signum, frame):
    global stop_signal
    if stop_signal is None:
        stop_signal = signum


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("output", type=Path, help="new JSONL output path")
    parser.add_argument("--items", type=int, default=10)
    parser.add_argument("--delay", type=float, default=1.0,
                        help="demo work seconds per item (0 to 60)")
    args = parser.parse_args()
    if args.items < 1:
        parser.error("--items must be positive")
    if not math.isfinite(args.delay) or not 0 <= args.delay <= 60:
        parser.error("--delay must be finite and between 0 and 60")

    signal.signal(signal.SIGTERM, request_stop)
    signal.signal(signal.SIGINT, request_stop)
    completed = 0
    try:
        with args.output.open("x", encoding="utf-8") as output:
            for item in range(1, args.items + 1):
                if stop_signal is not None:
                    break
                print(f"START item={item}", flush=True)
                time.sleep(args.delay)  # Replace with bounded, synchronous work.
                record = {"id": item, "square": item * item}
                output.write(json.dumps(record) + "\n")
                output.flush()
                os.fsync(output.fileno())
                completed += 1
                print(f"SAVED item={item}", flush=True)
    except OSError as exc:
        print(f"ERROR: {exc}", file=sys.stderr)
        return 1

    if stop_signal is not None:
        name = signal.Signals(stop_signal).name
        print(f"STOPPED signal={name} completed={completed}", flush=True)
        return 128 + stop_signal
    print(f"DONE completed={completed}", flush=True)
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Replace time.sleep and the square calculation with bounded, synchronous work, keeping the save steps after successful processing. The os.fsync documentation specifies flushing a buffered file before calling fsync(). Syncing every record adds overhead and suits this low-throughput demo; it is not a multi-file transaction or a guarantee of power-loss recovery on every filesystem.

The time.sleep documentation explains that sleep continues for the remaining duration after a signal when the handler does not raise. This demo therefore finishes the current simulated wait after a stop request.

The example explicitly chooses these exit codes: 0 for completion, 1 for file errors, 2 for argument errors, 143 after SIGTERM, and 130 after SIGINT. A stopped status does not mean the entire batch finished or that the process died from an unhandled signal.

4. Validation: signal a process during its first item

Start with a manual observation:

python3 graceful_batch.py graceful-demo/manual.jsonl --items 10 --delay 2

Press Ctrl+C after START item=1. You should see SAVED item=1, then STOPPED signal=SIGINT completed=1; pressing later may let more items finish.

To test SIGTERM, identify this Python process’s PID in another terminal, then replace 12345 with its actual PID:

kill -TERM 12345

For a precise check that the current record finishes and later records never start, save the following as verify_stop.py beside the script. It waits for the first item to start before sending a signal and cleans up its own child process on timeout.

import json
from pathlib import Path
import selectors
import signal
import subprocess
import sys
import tempfile

script = Path("graceful_batch.py").resolve()
with tempfile.TemporaryDirectory() as work:
    for sig in (signal.SIGTERM, signal.SIGINT):
        target = Path(work) / (sig.name + ".jsonl")
        process = subprocess.Popen(
            [sys.executable, str(script), str(target), "--delay", "2"],
            stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True,
        )
        try:
            with selectors.DefaultSelector() as ready:
                ready.register(process.stdout, selectors.EVENT_READ)
                assert ready.select(10), "no startup output"
            assert process.stdout.readline().strip() == "START item=1"
            process.send_signal(sig)
            stdout, stderr = process.communicate(timeout=10)
            assert process.returncode == 128 + sig, (stdout, stderr)
            assert "STOPPED" in stdout and "DONE" not in stdout
            rows = [json.loads(line) for line in target.read_text().splitlines()]
            assert rows == [{"id": 1, "square": 1}], rows
            print(f"PASS {sig.name}: saved one record, exit={process.returncode}")
        finally:
            if process.poll() is None:
                process.kill()
                process.communicate()

Run:

python3 verify_stop.py

Expected output:

PASS SIGTERM: saved one record, exit=143
PASS SIGINT: saved one record, exit=130

A failed check raises an exception instead of printing success. The test checks both the exit status and actual JSONL content; its temporary directory is removed afterward.

5. Troubleshooting: file errors and slow shutdown

Existing output or invalid arguments

Run these commands individually. They intentionally fail, so keep them out of a shell script that exits immediately on errors:

python3 graceful_batch.py graceful-demo/complete.jsonl --delay 0
echo "$?"
python3 graceful_batch.py graceful-demo/invalid.jsonl --delay nan
echo "$?"

The first command reports an existing file and returns 1, preserving the original three records. The second returns 2 without creating invalid.jsonl. A missing output directory also returns 1; create your target directory before running the program.

The process keeps waiting after a stop request

Check whether the current item is waiting on a network request, subprocess, or long calculation. Use shorter work units and timeouts; adding a signal handler does not impose a deadline on those operations. Repeated SIGTERM or SIGINT signals still only request a stop; this example does not force an interruption of the current record.

No stopped message after kill -9

SIGKILL cannot be caught, so cleanup and file closing cannot run. Forced termination, disk errors, or power loss can still leave an incomplete last line; do not treat this JSONL output as a reliable checkpoint.

A scheduled task treats stopping as success

Preserve nonzero exit statuses and distinguish completion, requested stopping, and execution failure in your logs. Fix the interpreter and working directory in your entry point as shown in the cron environment troubleshooting tutorial. This example does not change service or scheduler configuration, and it does not forward signals to separately launched child processes.

6. Summary

Keep the signal handler limited to recording a stop request, and check it before starting the next record. Finish the current calculation and save in the normal flow, then verify shutdown with real signals, exit statuses, and output files. Before using this pattern in a real job, bound each work unit’s duration and design checkpoint recovery separately.