Fix Delayed Python Logs with -u, flush, and Bash pipefail

A Python job is running, but its log stays quiet for seconds or minutes. Adding tee can make progress appear only when the script finishes. A separate problem is easy to miss: Python fails, yet the pipeline returns zero.

This tutorial uses a three-second demo to fix output buffering and pipeline status hiding failures. It targets beginner-to-intermediate readers running batch jobs, background tasks, and build scripts on Linux.

1. Prepare the environment

Use Linux, Python 3.9 or newer, Bash, and GNU Coreutils’ tee. No third-party Python packages are needed. The examples were tested on Linux with Python 3.11.4.

1. Create a scratch directory. Run all subsequent commands there:

python3 --version
command -v bash
tee --help
mkdir python-buffer-demo
cd python-buffer-demo
unset PYTHONUNBUFFERED

This clears PYTHONUNBUFFERED only in the current shell so you can reproduce default behavior. The demo’s tee commands overwrite existing logs; use a fresh directory when repeating the experiment.

2. Save the next section’s code as demo.py, then run python3 demo.py --help.

2. Reproduce delayed output in a pipeline

"""Demonstrate delayed stdout and an optional failure. Python 3.9+."""
import argparse
import sys
import time


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--fail", action="store_true", help="exit with status 7")
    args = parser.parse_args()
    print("START")
    for number in range(1, 4):
        print(f"processed={number}")
        time.sleep(1)
    if args.fail:
        print("ERROR: simulated failure", file=sys.stderr)
        return 7
    print("DONE")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Run it directly, then through a pipeline:

python3 demo.py
python3 demo.py 2>&1 | tee buffered.log

In an ordinary interactive terminal, the first command usually prints incrementally. The second usually delivers this small amount of stdout together after about three seconds. That is expected for this demo; it does not mean the script stopped working.

Python’s standard-stream documentation describes interactive stdout as normally line-buffered and non-interactive stdout as normally block-buffered. With a pipe, Python writes to the pipe rather than directly to the terminal. Here, 2>&1 combines stderr with stdout for recording.

3. Method 1: add -u to the launch command

Keep the code unchanged and run:

python3 -u demo.py 2>&1 | tee live.log

You should quickly see START and processed=1, followed by progress roughly once per second, and finally DONE. Compare the completed logs:

cmp buffered.log live.log
printf 'cmp_exit=%s\n' "$?"

Expect cmp_exit=0. The difference is arrival time, rather than the final content of this successful run.

Python’s command-line documentation defines -u as disabling stdout and stderr buffering; stdin is unaffected. Place it before the script filename. If your launcher makes environment settings easier, use the equivalent:

PYTHONUNBUFFERED=1 python3 demo.py 2>&1 | tee env.log
cmp live.log env.log

The variable applies only to that command. If a scheduled job also has interpreter, directory, or missing-variable errors, use the scheduled-job environment tutorial to fix its entry point before adding -u.

4. Method 2: flush selected progress messages

When you can edit the script, use flush=True for messages that need prompt delivery:

import time

print("START", flush=True)
time.sleep(1)
print("DONE", flush=True)

Save this as flush_demo.py and run:

python3 flush_demo.py 2>&1 | tee flush.log

Expect START first and DONE about a second later. Python’s print documentation specifies that flush=True forces the stream to flush. It affects that call; other ordinary print calls can still buffer.

Use -u to adjust an existing Python job’s standard output globally, or flush=True for selected checkpoints. Frequent flushing adds write overhead. Neither option synchronizes result files to disk or controls a downstream tool’s buffering. For persistent structured logs, see size-based log rotation.

5. Verify failures with Bash pipefail

Each bash -c below starts a separate Bash with an explicit option setting. Run these individually: the second intentionally returns nonzero, so do not place this demonstration inside an outer script that stops on errors.

bash -c 'set +o pipefail; python3 -u demo.py --fail 2>&1 | tee hidden-fail.log'
printf 'pipeline_exit=%s\n' "$?"

bash -c 'set -o pipefail; python3 -u demo.py --fail 2>&1 | tee visible-fail.log'
printf 'pipeline_exit=%s\n' "$?"

Both show progress and ERROR: simulated failure. With a successful tee, the first reports pipeline_exit=0 and the second pipeline_exit=7.

The Bash pipeline manual defines the default status as the last command’s status. With pipefail, it uses the rightmost nonzero status, or zero if all commands succeed. This does not itself stop later commands. If tee also fails, its status can determine the pipeline status; the result is not always Python’s exit code.

An automation wrapper can handle the status explicitly. Save this as run_demo.sh and run bash run_demo.sh in the current directory. Expect exit status 7:

#!/usr/bin/env bash
set -o pipefail

python3 -u demo.py --fail 2>&1 | tee run.log
job_status=$?
if [ "$job_status" -ne 0 ]; then
    printf 'FAILED: pipeline exit=%s\n' "$job_status" >&2
    exit "$job_status"
fi
printf 'SUCCESS\n'

Remove --fail and rerun: it should then print SUCCESS and return zero. This covers the demo; real jobs also need application-level output checks. For launching tools from Python with a timeout, combine it with the subprocess timeout and logging tutorial.

6. Troubleshooting

The GNU tee manual documents replacement by default and appending with -a. Choose deliberately: appending can mix multiple runs.

If filters or a remote log platform follow the Python process, check their flush policies too. Prompt logs alone do not prove correctness or guarantee that a process will finish.

7. Summary

Use a short demo to establish whether Python buffering causes the delay, then choose -u or flush=True for key messages. When recording through tee, enable Bash’s pipefail and handle the status. Verify progress timing, failure status, and final application output before adding the command to automation.