Find Duplicate Files with Python: Size Filters, Chunked SHA-256, and Read-Only Reports

Downloads, backup copies, and batch outputs accumulate under different names even when their contents match. Comparing sizes alone groups unrelated files together, while reading whole files into memory makes large-file inspection expensive.

This tutorial builds a read-only duplicate-file reporting tool: filter by byte size, hash in chunks with SHA-256, and emit JSON. It does not delete, move, or rewrite inspected files. It targets beginner-to-intermediate readers comfortable with terminal commands; use the large-file ranking tool first if you need to narrow the search.

1. Method: filter by size, then hash contents

1. Traverse a local directory, including hidden files. Use lstat() and the regular-file type check to skip symlinks and special files such as FIFOs.
2. Keep one path per (st_dev, st_ino) pair so hardlinks are not treated as independent copies; see the Python stat result reference for these fields.
3. Group regular files by st_size. Read contents only for size buckets containing at least two independent files.
4. Read 1 MiB at a time and update SHA-256. Files with matching sizes and digests form a candidate duplicate group.

The comparison concerns raw bytes. Different line endings, encodings, or CSV row orders affect the result. Empty files can also form groups. Hash grouping is a filter; compare candidates byte by byte when strict confirmation is required.

2. Prepare the environment and fixtures

Use Linux, Python 3.9 or newer, and Bash or Zsh. Only the standard library is needed; there are no third-party packages to install. Start in a new demo directory:

python3 --version
mkdir duplicate-demo
cd duplicate-demo
python3 - <<'PY_DATA'
from pathlib import Path
root = Path("data")
root.mkdir()
(root / "a.txt").write_bytes(b"alpha\n")
(root / "b.txt").write_bytes(b"alpha\n")
(root / "c.txt").write_bytes(b"bravo\n")
(root / "unique.txt").write_bytes(b"one longer unique record\n")
(root / "empty-a").write_bytes(b"")
(root / "empty-b").write_bytes(b"")
PY_DATA

a.txt and b.txt match. c.txt is also 6 bytes long but has different contents. Two empty files exercise the zero-byte bucket. Keep the script and reports outside data to avoid scanning the report itself. Use a fresh directory for another run.

3. Complete implementation

Save the following as find_duplicates.py in the current duplicate-demo directory:

"""Find equal-size, equal-SHA-256 file groups. Linux, Python 3.9+."""
import argparse
from collections import defaultdict
import hashlib
import json
import os
from pathlib import Path
import stat
import sys


def sha256_file(path):
    digest = hashlib.sha256()
    with path.open("rb") as handle:
        while True:
            block = handle.read(1024 * 1024)
            if not block:
                break
            digest.update(block)
    return digest.hexdigest()


def find_groups(root):
    def fail(error):
        raise error

    sizes = defaultdict(list)
    seen = set()
    for folder, dirs, names in os.walk(root, onerror=fail, followlinks=False):
        dirs.sort()
        for name in sorted(names):
            path = Path(folder) / name
            info = path.lstat()
            if not stat.S_ISREG(info.st_mode):
                continue
            identity = (info.st_dev, info.st_ino)
            if identity in seen:
                continue
            seen.add(identity)
            sizes[info.st_size].append(path)

    groups = []
    for size, paths in sorted(sizes.items()):
        if len(paths) < 2:
            continue
        hashes = defaultdict(list)
        for path in paths:
            hashes[sha256_file(path)].append(path.relative_to(root).as_posix())
        for digest, matches in sorted(hashes.items()):
            if len(matches) > 1:
                groups.append({"size_bytes": size, "sha256": digest,
                               "files": sorted(matches)})
    return groups


def main():
    parser = argparse.ArgumentParser(description=__doc__, allow_abbrev=False)
    parser.add_argument("directory", type=Path, help="quiet local directory to scan")
    args = parser.parse_args()
    try:
        if args.directory.is_symlink():
            raise ValueError("scan root must not be a symlink")
        root = args.directory.resolve(strict=True)
        if not root.is_dir():
            raise ValueError("scan root must be a directory")
        groups = find_groups(root)
    except (OSError, ValueError) as exc:
        print(f"ERROR: {exc}", file=sys.stderr)
        return 1
    print(json.dumps({"groups": groups}, ensure_ascii=True, indent=2))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

The official os.walk() reference documents that traversal errors are ignored by default. Here, onerror=fail aborts immediately so a missed directory cannot masquerade as “no duplicates”; followlinks=False avoids descending into linked directories. File entries still need a type check to exclude file symlinks and broken links.

hashlib.update() accepts successive byte chunks with the same effect as their concatenation. The code does not retain entire file contents. Paths, the inode set, and grouping structures still use memory proportional to the number of files, so total memory is not constant.

Traversal sorts directory and file names; the first encountered path represents a hardlinked file. Results sort by size and digest, with sorted paths inside each group. JSON uses ASCII escaping: Chinese filenames appear as \u... but decode to the original names, and paths containing special characters can be stored too.

4. Run and validate the report

python3 find_duplicates.py --help
python3 find_duplicates.py data > report.json
python3 - <<'PY_CHECK'
import json
from pathlib import Path
report = json.loads(Path("report.json").read_text(encoding="utf-8"))
assert [g["files"] for g in report["groups"]] == [
    ["empty-a", "empty-b"], ["a.txt", "b.txt"]
]
assert [g["size_bytes"] for g in report["groups"]] == [0, 6]
print("OK: two groups; same-size c.txt excluded")
PY_CHECK

Expected output:

OK: two groups; same-size c.txt excluded

The report contains a groups array. Each group has size_bytes, sha256, and a relative-path list named files. No candidate duplicates produces {"groups": []}. Exit code 0 means the scan completed, including when duplicates exist. Exit code 1 means scanning or reading failed, with no JSON on standard output. argparse returns 2 for invalid command arguments.

To track whether a whole input set changes between runs, use a SHA-256 file manifest. That checks paths and versions; this tutorial looks for candidate copies with matching contents within a single scan.

5. Verify hardlinks, symlinks, and failures

Run this check once in the same new demo directory. It creates links and a FIFO only alongside the fixtures in data:

python3 - <<'PY_LINKS'
import json
import os
from pathlib import Path
import subprocess
import sys

root = Path("data")
os.link(root / "a.txt", root / "z-hardlink.txt")
(root / "file-link").symlink_to("a.txt")
(root / "broken-link").symlink_to("missing.txt")
(root / "loop").symlink_to(".", target_is_directory=True)
os.mkfifo(root / "pipe")
result = subprocess.run([sys.executable, "find_duplicates.py", "data"],
                        capture_output=True, text=True, check=True)
report = json.loads(result.stdout)
assert [g["files"] for g in report["groups"]] == [
    ["empty-a", "empty-b"], ["a.txt", "b.txt"]
]
failed = subprocess.run([sys.executable, "find_duplicates.py", "missing-dir"],
                        capture_output=True, text=True, check=False)
assert failed.returncode == 1 and failed.stdout == ""
assert "ERROR:" in failed.stderr
print("OK: hardlinks, symlinks, FIFO, missing directory")
PY_LINKS

Expected output:

OK: hardlinks, symlinks, FIFO, missing directory

z-hardlink.txt and a.txt share an inode, so this does not represent an extra reclaimable copy. File symlinks, broken links, directory-loop symlinks, and FIFOs do not enter the report. Use the broken-symlink checker when you need to investigate link targets separately.

Shell redirection creates or truncates a report before Python starts. Check the exit status before consuming it:

if python3 find_duplicates.py missing-dir > failed-report.json; then
  printf 'Unexpected success\n'
else
  printf 'Scan failed; do not use failed-report.json\n'
fi

This failure example leaves an empty failed-report.json; it is not a valid empty report. In an actual workflow, let the caller accept output only after the command succeeds, and keep the report outside the scanned directory.

6. Troubleshooting and limits

Stop writers before scanning. This script is not a filesystem snapshot. Concurrent changes between discovery and reading can cause missed matches, incorrect grouping, or errors; it is not designed for hostile path replacement. Reading may also update filesystem access timestamps.

Matching digests do not establish that a file can be deleted. Backup roles, permissions, sparse files, compression, copy-on-write, and hardlinks affect retention or actual disk use. This tool does not estimate reclaimable space; confirm business requirements and compare candidate bytes again before deletion.

7. Summary

Filter out impossible matches by size, then hash in chunks to avoid unnecessary reads and whole-file memory allocation. Validate equal-size mismatches, hardlinks, and failure paths before generating a report for a stable directory. Leave candidate groups for human review or a later business process, preserving the report and exit status.