Rotate Python Logs by Size with RotatingFileHandler

An automation script that keeps appending to task.log can eventually produce a file too large to inspect and consume valuable disk space. This tutorial uses the standard-library RotatingFileHandler to rotate by size, retain a fixed number of backups, and verify that a failed job records a traceback and returns a nonzero exit code.

The example is for beginner-to-intermediate readers running batch scripts with one writing process at a time. If the server is already full, use the Linux deleted-file disk-space troubleshooting steps to recover space before adding rotation.

1. Choose a method: let Python manage its own log

Python owns rotation here. The active file is task.log, the newest backup is task.log.1, and older backups are .2 and .3. Older backups are replaced once the retention count is reached.

The official Python handler documentation states that both maxBytes and backupCount must be nonzero to enable rotation. We use 4096 bytes and three backups to make it easy to observe; real jobs can use a larger threshold.

Choose one owner for rotation; avoid having another rotation tool rename these same files. This handler only handles application records sent through this logger. It does not automatically capture print() or subprocess output.

2. Prepare the environment

You need Python 3.9 or newer and no third-party packages. The example was tested on Linux with Python 3.11.4. The shell commands work in POSIX shells such as bash and zsh.

1. Create a fresh demo directory. For another experiment, use a new directory name to avoid mixing in old logs:

python3 --version
mkdir rotating-log-demo
cd rotating-log-demo

2. Save the complete code in the next section as rotate_job.py.

3. Complete implementation and usage

"""Rotate one Python process's application log. Standard library only."""
import argparse
import logging
from logging.handlers import RotatingFileHandler
from pathlib import Path
import sys


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--log-dir", type=Path, default=Path("logs"))
    parser.add_argument("--count", type=int, default=200)
    parser.add_argument("--max-bytes", type=int, default=4096)
    parser.add_argument("--backups", type=int, default=3)
    parser.add_argument("--fail", action="store_true")
    args = parser.parse_args()
    if args.count < 0 or args.max_bytes <= 0 or args.backups <= 0:
        parser.error("count must be nonnegative; max-bytes and backups must be positive")

    log_path = args.log_dir / "task.log"
    try:
        args.log_dir.mkdir(parents=True, exist_ok=True)
        handler = RotatingFileHandler(
            log_path, maxBytes=args.max_bytes, backupCount=args.backups,
            encoding="utf-8",
        )
    except OSError as exc:
        print(f"ERROR: cannot open log: {exc}", file=sys.stderr)
        return 1

    handler.setFormatter(logging.Formatter(
        "%(asctime)s %(levelname)s %(message)s", datefmt="%Y-%m-%dT%H:%M:%S",
    ))
    logger = logging.getLogger("rotation_demo")
    logger.setLevel(logging.INFO)
    logger.propagate = False
    logger.addHandler(handler)
    try:
        logger.info("START count=%d", args.count)
        # Replace this loop with your synchronous application work.
        for item in range(args.count):
            logger.info("item=%04d status=ok payload=%s", item, "x" * 120)
        if args.fail:
            raise RuntimeError("demo job failed")
        logger.info("DONE count=%d", args.count)
    except Exception:
        logger.exception("FAILED")
        print(f"FAILED: see {log_path}", file=sys.stderr)
        return 2
    finally:
        logger.removeHandler(handler)
        handler.close()
    print(f"OK: log={log_path} count={args.count}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

The script creates the directory before configuring UTF-8 logging. Setting propagate = False keeps the root logger’s handlers from also emitting the same record. Configure the handler once at the entry point and close it when finished; do not add it on every loop iteration.

1. Read the help and write 200 demo records:

python3 rotate_job.py --help
python3 rotate_job.py --count 200
ls -lh logs/task.log*
tail -n 2 logs/task.log

The script invocation prints:

OK: log=logs/task.log count=200

You should see task.log and the three backups .1, .2, and .3. The active file ends with DONE count=200. Timestamps use the machine’s local time; inspect the actual file sizes rather than relying on fixed output values.

2. Replace the demo message loop with your own work, keeping start, completion, and exception records. For external tools, use the subprocess timeout and logging tutorial to capture their output separately. Redirecting output to an ordinary file does not activate this rotating handler.

4. Verify file count and retained content

Save this code as check_rotation.py and run python3 check_rotation.py in the same directory:

from pathlib import Path

folder = Path("logs")
files = sorted(folder.glob("task.log*"))
assert [p.name for p in files] == [
    "task.log", "task.log.1", "task.log.2", "task.log.3",
]
# These demo messages are short ASCII records, not arbitrary application data.
assert all(0 < p.stat().st_size <= 4096 for p in files)
assert "DONE count=200" in (folder / "task.log").read_text(encoding="utf-8")
for path in files:
    print(f"{path.name}: {path.stat().st_size} bytes")
print("PASS: one active log, three backups, latest completion retained")

The last line should be:

PASS: one active log, three backups, latest completion retained

Run python3 rotate_job.py --count 200 and python3 check_rotation.py again. Confirm that only these four files remain and that the latest completion is in the active file. The size assertion applies to this demo’s short ASCII messages. Oversized individual messages, multibyte encoding, and similar cases can exceed the threshold; maxBytes is not a hard disk quota.

Rotation discards older records, so it does not fit audit data that must be retained permanently. Archive that data separately. Giving every run a new filename also requires a separate cleanup policy.

5. Verify failures and adjust retention

1. Run these commands individually. The failure demo deliberately returns 2, so do not put it directly into a validation script that exits on the first error:

python3 rotate_job.py --count 2 --fail
printf 'exit=%s\n' "$?"
tail -n 8 logs/task.log

Expect FAILED: see logs/task.log and exit=2 in the terminal. The log tail includes FAILED, Traceback, and RuntimeError: demo job failed. A successful call returns 0; failure to create the directory or open the file returns 1; argparse returns 2 for invalid parameters.

2. After testing the small threshold, try 10 MiB and five backups in the same directory:

python3 rotate_job.py --count 200 --max-bytes 10485760 --backups 5

This means one active file and up to five backups; it does not mean five days of retention. Records already discarded by the smaller policy cannot be recovered. About 60 MiB is only a capacity estimate for short records, not a strict limit. Reducing the backup count also does not immediately remove every existing higher-numbered file.

Use absolute paths for the interpreter, script, and --log-dir when connecting an existing automation entry point, and preserve the script’s exit code. Allow only one writing process for a given log path. For concurrent work, the official Python multiprocessing logging guide recommends sending records through a queue or socket to one writer.

6. Troubleshooting and direct fixes

I/O errors during writes or renames may be handled by logging’s handleError instead of automatically failing the job. Monitor stderr, disk space, and actual log content together. This example’s exit code is not a log-durability guarantee.

Search both the active file and backups when investigating errors. The log triage tutorial shows how to narrow results with rg. Backup numbers indicate relative age; do not inspect only the largest number.

Summary

Configure a nonzero size threshold and backup count for a single-process Python script to apply a clear retention policy. Test rotation, repeated runs, and failure records with a small threshold before choosing a practical capacity. Concurrent writers and permanent archives need their own design, and checking file count does not validate business results.