Python TemporaryDirectory: Clean Up Workspaces and Keep Finished Reports
On this page
A batch script fails after writing intermediate files and leaves its workspace behind. Another common mistake is putting the final report inside that workspace, so cleanup deletes the result too. Use TemporaryDirectory to give each run a private workspace, clean it when the scope ends, and keep successful output outside it.
This beginner-to-intermediate tutorial uses Linux and Python 3.10 or newer, with no third-party packages. It was tested with Python 3.11.4. The runnable task builds a text report: remove blank lines and surrounding whitespace, number the remaining lines, and verify that failures preserve the previous report.
1. Method: separate temporary files from final output
- Leave the input in place and read it without modification.
- Create a randomly named
.report-*workspace inside the output directory and write a candidate report there. - Close the candidate report, then use
os.replace()to put it at the final path outside the workspace. - Leave the
withblock and clean up this run’s workspace.
The official TemporaryDirectory documentation describes its context-manager cleanup. We set dir= to choose its location and prefix= to identify its purpose; the parent must already exist and be writable. The default location can depend on the environment and is not necessarily /tmp.
Keeping the workspace inside the final directory places the candidate and destination on the same filesystem. A successful os.replace() atomically replaces the path, overwriting an existing destination file. It does not serialize multiple writers. This example assumes a trusted local directory and one writer per destination; it does not provide durability across a power failure.
2. Set up the environment and input
- Run these commands in one terminal. Choose another directory name when repeating the exercise to preserve previous results:
python3 --version
mkdir tempfile-report-demo
cd tempfile-report-demo
mkdir output
python3 - <<'PY_DATA'
from pathlib import Path
Path("notes.txt").write_text(" first task \n\n第二个任务\n", encoding="utf-8")
PY_DATA
- Save the next section as
task_report.pyin the current directory. Run all subsequent commands insidetempfile-report-demo.
3. Complete implementation
"""Build a numbered UTF-8 report in a private temporary workspace."""
import argparse
import os
from pathlib import Path
import sys
import tempfile
def build_report(source, output, fail=False):
source = source.resolve(strict=True)
if not source.is_file():
raise ValueError("input must be a regular file")
output = output.parent.resolve(strict=True) / output.name
if output.is_symlink():
raise ValueError("output must not be a symlink")
if output.exists() and output.samefile(source):
raise ValueError("input and output must be different files")
with tempfile.TemporaryDirectory(prefix=".report-", dir=output.parent) as name:
work = Path(name)
print(f"WORK={work}", file=sys.stderr, flush=True)
staged = work / "report.txt"
count = 0
with source.open(encoding="utf-8") as incoming:
with staged.open("x", encoding="utf-8", newline="\n") as outgoing:
for line in incoming:
text = line.strip()
if text:
count += 1
outgoing.write(f"{count}\t{text}\n")
if fail:
raise RuntimeError("simulated failure before replacement")
os.replace(staged, output)
return count
def main():
parser = argparse.ArgumentParser(description=__doc__, allow_abbrev=False)
parser.add_argument("source", type=Path, help="existing UTF-8 text file")
parser.add_argument("output", type=Path, help="report path; replaces an existing file")
parser.add_argument("--fail", action="store_true", help="fail before replacement")
args = parser.parse_args()
try:
count = build_report(args.source, args.output, args.fail)
except (OSError, UnicodeError, ValueError, RuntimeError) as exc:
print(f"ERROR: {exc}", file=sys.stderr)
return 1
print(f"OK: {count} nonempty lines")
return 0
if __name__ == "__main__":
raise SystemExit(main())
The file context managers close before replacement, and the temporary-directory context ends afterward. --fail deliberately raises an exception before replacement to exercise cleanup. Invalid UTF-8 follows the same error-handling path. The script rejects input and output pointing to the same file, and rejects output symlinks.
A successful run replaces the specified output file. Empty or whitespace-only input successfully produces an empty file. Reading is line by line rather than loading the entire input, although a single very long line still consumes memory.
4. Run and verify successful output
python3 task_report.py --help
python3 task_report.py notes.txt output/report.txt
cat output/report.txt
Standard error shows WORK=.../output/.report-random-characters to identify this run’s workspace. The success message on standard output and the report contents are:
OK: 2 nonempty lines
1 first task
2 第二个任务
Afterward, output/report.txt remains and the .report-* directory should be gone. The report was moved outside the workspace, so workspace cleanup cannot remove it. Exit codes are 0 for success, 1 for reading, writing, or simulated failures, and 2 for command-line argument errors.
Save the following as verify_report.py to check normal completion, failure before replacement, bad encoding, and isolated concurrent workspaces:
"""Verify cleanup and output preservation without leaving test data."""
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
import re
import subprocess
import sys
import tempfile
script = Path("task_report.py").resolve(strict=True)
with tempfile.TemporaryDirectory(prefix="check-", dir=Path.cwd()) as name:
root = Path(name)
source = root / "notes.txt"
output = root / "report.txt"
source.write_text(" first task \n\n第二个任务\n", encoding="utf-8")
def run(target, *options):
result = subprocess.run(
[sys.executable, str(script), str(source), str(target), *options],
capture_output=True, text=True, encoding="utf-8", timeout=10,
)
match = re.search(r"^WORK=(.+)$", result.stderr, re.MULTILINE)
if match:
assert not Path(match[1]).exists(), result.stderr
return result
result = run(output)
assert result.returncode == 0, result.stderr
assert output.read_text(encoding="utf-8") == "1\tfirst task\n2\t第二个任务\n"
previous = output.read_bytes()
result = run(output, "--fail")
assert result.returncode == 1 and not result.stdout
assert output.read_bytes() == previous
result = run(root / "never-created.txt", "--fail")
assert result.returncode == 1
assert not (root / "never-created.txt").exists()
source.write_bytes(b"valid\n\xffinvalid\n")
result = run(output)
assert result.returncode == 1 and output.read_bytes() == previous
source.write_text("ready\n", encoding="utf-8")
assert run(source).returncode == 1
assert run(output, "--fai").returncode == 2
with ThreadPoolExecutor(max_workers=4) as pool:
results = list(pool.map(run, [root / f"report-{i}.txt" for i in range(4)]))
assert all(result.returncode == 0 for result in results)
paths = [re.search(r"^WORK=(.+)$", r.stderr, re.MULTILINE)[1] for r in results]
assert len(set(paths)) == 4
assert not list(root.glob(".report-*"))
print("PASS: cleanup, failure preservation, UTF-8 errors, isolated workspaces")
Run:
python3 verify_report.py
Expect PASS: cleanup, failure preservation, UTF-8 errors, isolated workspaces. The checks verify unchanged bytes in the old report, no newly created report on failure, and removed workspaces. Test inputs and reports are also cleaned up at the end. Concurrent checks use different output files: unique workspaces do not protect a shared destination from concurrent writes.
5. Finish external commands before closing the workspace
When an external tool uses the workspace, wait for it with synchronous subprocess.run(). This standalone example waits for a child to write a file, checks it, and then removes the directory:
python3 - <<'PY_CHILD'
from pathlib import Path
import subprocess
import sys
import tempfile
with tempfile.TemporaryDirectory(prefix="child-", dir=Path.cwd()) as work:
subprocess.run(
[sys.executable, "-c",
"from pathlib import Path; Path('intermediate.txt').write_text('done', encoding='utf-8')"],
cwd=work, check=True, timeout=10,
)
assert (Path(work) / "intermediate.txt").read_text(encoding="utf-8") == "done"
assert not Path(work).exists()
print("PASS: child finished before cleanup")
PY_CHILD
Starting a background process and immediately leaving with can remove paths it still needs. For real commands, continue with Python subprocess timeouts and logs.
6. Troubleshooting and limits
| Symptom | Cause and action |
|---|---|
| Missing parent directory | Run mkdir -p output first; dir= does not create its parent. |
| Old report unchanged after failure | Expected for failures before replacement; fix the input or task error and rerun. |
| Exit 1 but the report was updated | Cleanup can fail after replacement; inspect stderr and the WORK path, then check the output and leftover workspace separately. |
| Permission denied | Check output-directory write permissions and close file handles. Do not hide cleanup failures with ignore_cleanup_errors=True. |
| No space left on device | Temporary files consume space too. Add a disk-space preflight and budget for old and new reports coexisting. |
| Workspace remains after termination | Forced termination, default SIGTERM handling, or power loss can bypass Python cleanup. Confirm the task and its children have stopped before handling the exact recorded directory. |
SIGKILL cannot be caught, blocked, or ignored, so with cleanup cannot cover every termination mode. Do not clear all of /tmp or every .report-* directory: another task may still be using its workspace.
This example preserves the old report on failures before replacement. It does not call fsync() and does not establish power-loss durability. For important JSON state, see the flush order and limits in atomic JSON writes with Python.
7. Summary
Use TemporaryDirectory to keep intermediate files inside a workspace owned by one run and clean it when its scope ends. Finish writing the final result before moving it outside that workspace, and test both success and failure paths. Wait for external commands to finish; handle leftovers from forced termination by their recorded ownership.
- Author: yanzhi
- Original article: https://www.bobobk.com/en/python-tempfile-workspace-cleanup.html
- License: This work is licensed under CC BY-NC-ND 4.0. See the license for attribution, noncommercial use and no-derivatives terms.
Related articles
- Test Directory Writes with Python: Permissions, Read-Only Storage, and Path Errors
- Validate Python Environment Variables: Booleans, Empty Values, and Integers
- Create Reproducible gzip Files with Python: Control Headers and Verify Hashes
- Build URL Query Parameters in Python: Unicode, Plus Signs, and Repeated Keys
- Fix Duplicate Python Logs: Repeated Handlers and Logger Propagation