Debug a Stuck Python Script with faulthandler Thread Dumps

A batch script stops producing logs, but its process is still running. Restarting loses the evidence: is it waiting for a lock, a network operation, or a worker thread? Use Python’s standard-library faulthandler to capture where threads are executing, then inspect the waiting calls.

The example needs Linux and a regular GIL-enabled CPython 3.10+ build, with no pip packages. It was tested on Linux with Python 3.11.4. The intentional lock wait ends after the configured duration, so the exercise does not hang forever.

1. Choose a diagnostic method

• Method 1: timed snapshots. Configure them before work starts to record stacks periodically during unattended jobs.
• Method 2: on-demand snapshots. Register SIGUSR1 beforehand and trigger a dump from another terminal when investigating a Linux process.

The official Python documentation describes the timer and cancellation interfaces. This example uses exit=False to preserve the running process. python3 -X faulthandler enables fault-signal handling; it does not schedule snapshots or register SIGUSR1 for you.

If the job never starts, first check its interpreter and working directory with the cron environment troubleshooting guide.

2. Set up and save the complete example

1. Create a practice directory and check the interpreter:

mkdir python-hang-demo
cd python-hang-demo
python3 --version

2. Download hang_demo.py into that directory, or save this complete code under the same name:

"""Inspect a bounded lock wait using CPython 3.10+ on Linux."""
import argparse
import faulthandler
import os
import signal
import threading
from pathlib import Path


def worker(lock):
    with lock:
        print("WORKER: acquired lock", flush=True)


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--log", type=Path, default=Path("stacks.log"))
    parser.add_argument("--seconds", type=int, default=8)
    parser.add_argument("--interval", type=int, default=2)
    args = parser.parse_args()
    if not (1 <= args.seconds <= 3600 and 1 <= args.interval <= 3600):
        parser.error("seconds and interval must be between 1 and 3600")

    lock = threading.Lock()
    # Exclusive creation preserves stack logs from previous runs.
    with args.log.open("x", encoding="utf-8") as stack_log:
        faulthandler.register(signal.SIGUSR1, file=stack_log, all_threads=True)
        try:
            faulthandler.dump_traceback_later(
                args.interval, repeat=True, file=stack_log, exit=False
            )
            with lock:
                thread = threading.Thread(target=worker, args=(lock,))
                thread.start()
                # Registration is complete before these readiness messages.
                print(f"READY pid={os.getpid()}", flush=True)
                print(f"SNAPSHOT: kill -USR1 {os.getpid()}", flush=True)
                thread.join(timeout=args.seconds)
            # Releasing the lock lets the waiting worker finish.
            thread.join()
        finally:
            faulthandler.cancel_dump_traceback_later()
            faulthandler.unregister(signal.SIGUSR1)
    print(f"DONE: {args.log}", flush=True)


if __name__ == "__main__":
    main()

3. Inspect the command options:

python3 hang_demo.py --help

--seconds controls how long the main thread waits while holding the lock in this demo; --interval controls the snapshot interval. Both accept 1–3600 seconds. Run the code as a separate practice process. Before integrating it into an existing service, check whether that service already uses SIGUSR1.

3. Method 1: record waiting calls periodically

python3 hang_demo.py --log timer.log --seconds 8 --interval 2
cat timer.log

The terminal first prints READY and SNAPSHOT. About eight seconds later, it prints WORKER: acquired lock and DONE: timer.log. The log contains Timeout (0:00:02)! and thread stacks; the exact snapshot count can vary at timing boundaries.

Look for two locations: the worker stops at with lock: inside worker, while the main thread is at thread.join(timeout=args.seconds), usually underneath waiting frames in threading.py. As the Thread.join documentation explains, a timeout ends that wait without stopping the thread. This demo then leaves with lock, releasing the lock so the worker can continue.

The main thread is holding a lock while waiting for a worker that needs it. In real code, move join() outside the locked section rather than simply increasing the wait. This corrected example runs independently:

import threading

lock = threading.Lock()

def worker():
    with lock:
        print("WORKER: acquired lock")

thread = threading.Thread(target=worker)
thread.start()
thread.join(timeout=2)
assert not thread.is_alive(), "worker did not finish"
print("OK: joined without holding the lock")

4. Method 2: dump stacks immediately with SIGUSR1

1. In terminal A, run this from the practice directory:

python3 hang_demo.py --log signal.log --seconds 30 --interval 60

2. Wait for READY, copy the complete kill -USR1 command printed by the program, and execute it in terminal B. Then read the log from the same practice directory:

cat signal.log

The interval is 60 seconds, while the demo ends after about 30 seconds, so the snapshot in this file comes from the manual signal. The process continues afterward, and terminal A eventually prints DONE: signal.log.

Send the signal only to the still-running practice process that printed the command. SIGUSR1 is a user-defined signal; its behavior depends on the receiving process. Without a registered handler, it is not a general command for inspecting Python stacks. This tutorial cannot inject a handler into an already stuck, uninstrumented process.

5. Integrate into a job and validate

Keep the file opening, signal registration, timer setup, and finally cleanup from the demo. Replace the artificial lock wait with your actual work function. Keep the log handle open throughout diagnostics, in this order: cancel the timer → unregister the signal → close the file.

Choose a new stack file for every run. Manage ordinary application logs with the log rotation tutorial, but do not directly rotate this registered handle. The file descriptor documentation requires configuring diagnostic output again after replacing the file.

Check that the log is nonempty, includes the worker function and a main-thread waiting frame, and that the job eventually exits successfully. If another program is stuck, instrument that process: the Python parent’s stack only shows its own wait. Use the subprocess timeout and logging tutorial to manage external commands.

6. Troubleshooting

This tutorial inspects Python thread stacks, without expanding native-library internals or profiling performance. Configure one periodic dump per process so modules do not overwrite each other’s timer. Repeated snapshots add log data; choose an interval for the job’s duration and manage historical files.

7. Summary

Configure timed snapshots or SIGUSR1 before a long job starts so a stall leaves evidence of its waiting calls. Verify the logs with the bounded lock exercise, then instrument your own script. Fix the lock scope or set suitable I/O timeouts after identifying the blocking call; dumping stacks does not release a wait.