Check Filename Case and Unicode Collisions with Python Before Copying Projects

After moving a Linux project to another computer, README.md and readme.md may no longer work as two independent files. Two visually identical café.txt names can also use different Unicode sequences. Checking names before copying is easier than investigating missing files afterward.

This beginner-to-intermediate tutorial builds a read-only filename preflight with Python’s standard library: find sibling names equivalent under an explicit policy, print JSON, and gate later automation with exit codes. It does not rename files, delete data, or change Hugo configuration.

1. Method: define a conservative comparison policy

1. Normalize each name to NFC, apply casefold(), then normalize to NFC again to produce a comparison key.
2. Compare immediate siblings only, including files, directories, hidden entries, and symlink names. Recurse into real subdirectories, not directory symlinks.
3. Report multiple names sharing a key as one candidate group. Identical names in different directories are not collisions. Conflicting parent directory names are reported, but their descendants are not compared as a merged tree.

The official Unicode documentation explains NFC composition of canonically equivalent sequences. The official casefold() documentation describes broader processing than lowercasing, including mapping ß to ss. This is a conservative tutorial policy, not an exact simulation of a particular filesystem; some findings need human review.

It does not check Windows reserved names, trailing dots, invalid characters, path lengths, visual confusables, or duplicate url values in Hugo front matter. Zero findings do not guarantee portability. For a site release, also run the Hugo internal link and image checker.

2. Setup: create an isolated fixture

Use Linux, Python 3.9 or newer, and bash/zsh; no dependencies are needed. Commands were tested with Python 3.11.4. The fixture requires a case-sensitive filesystem that preserves both Unicode spellings independently. Use a fresh demo directory when repeating it to avoid overwriting old fixtures.

python3 --version
mkdir filename-audit-demo
cd filename-audit-demo
python3 - <<'PY_DATA'
from pathlib import Path
root = Path("sample")
(root / "assets").mkdir(parents=True)
for name in ["README.md", "readme.md"]:
    (root / name).write_text(name + "\n", encoding="utf-8")
for name in ["caf\u00e9.txt", "cafe\u0301.txt"]:
    (root / "assets" / name).write_text("demo\n", encoding="utf-8")
assert len(list(root.glob("*.md"))) == 2, "use a case-sensitive demo filesystem"
assert len(list((root / "assets").iterdir())) == 2, "Unicode names were merged"
PY_DATA

Both assertions should pass. If either fails, stop and choose a demo filesystem supporting these independent names; do not manufacture conflicts in your real project.

3. Implementation: scan names without reading file contents

Save this complete code as check_names.py in the current directory, or download the script.

"""Read-only sibling filename collision preflight. Python 3.9+."""
import argparse
import json
import os
from pathlib import Path
import sys
import unicodedata


def name_key(name):
    return unicodedata.normalize(
        "NFC", unicodedata.normalize("NFC", name).casefold()
    )


def audit(root):
    if not root.is_dir():
        raise ValueError("root must be an existing directory")
    groups = []
    pending = [root]
    while pending:
        directory = pending.pop()
        by_key = {}
        with os.scandir(directory) as entries:
            for entry in entries:
                by_key.setdefault(name_key(entry.name), []).append(entry.name)
                if entry.is_dir(follow_symlinks=False):
                    pending.append(Path(entry.path))
        for names in by_key.values():
            if len(names) > 1:
                groups.append({
                    "directory": directory.relative_to(root).as_posix(),
                    "names": sorted(names),
                })
    groups.sort(key=lambda item: (item["directory"], item["names"]))
    return {"collision_groups": len(groups), "groups": groups}


def main():
    cli = argparse.ArgumentParser(description=__doc__, allow_abbrev=False)
    cli.add_argument("root", type=Path, help="quiet local directory to scan")
    args = cli.parse_args()
    try:
        report = audit(args.root)
    except (OSError, ValueError) as exc:
        print(f"ERROR: {exc}", file=sys.stderr)
        return 2
    print(json.dumps(report, ensure_ascii=True, sort_keys=True))
    return 1 if report["collision_groups"] else 0


if __name__ == "__main__":
    raise SystemExit(main())

The official os.scandir() documentation describes directory entries and is_dir(follow_symlinks=False). The context manager closes each iterator. Directory listing and type-query errors are not ignored: an incomplete scan does not produce an apparently successful report.

Scan a trusted, quiet local directory. The supplied root resolves normally and may itself be a directory symlink; child symlinks are not traversed. This is not an atomic snapshot or protection against directories being replaced during scanning. Mount points inside real subdirectories can be traversed. Memory depends mainly on pending directories, entries in one directory, and the size of the findings.

4. Usage: report first, then review manually

1. Show help, then scan the fixture. Do not put the last two commands in a script using set -e: exit 1 is expected here.

python3 check_names.py --help
python3 check_names.py sample
printf 'exit=%s\n' "$?"

Expected output:

{"collision_groups": 2, "groups": [{"directory": ".", "names": ["README.md", "readme.md"]}, {"directory": "assets", "names": ["cafe\u0301.txt", "caf\u00e9.txt"]}]}
exit=1

JSON uses ASCII escapes to distinguish combining characters: \u0301 is a combining accent and \u00e9 is precomposed. Original names in the report identify actual paths; do not use normalized comparison keys to overwrite filenames.

2. After deciding that both files should remain, rename only the demo files. For real projects, back up first, compare contents, and update references together. The SHA-256 dataset manifest tutorial provides content evidence.

mv sample/readme.md sample/notes.md
python3 - <<'PY_FIX'
from pathlib import Path
source = Path("sample/assets/cafe\u0301.txt")
target = source.with_name("cafe-notes.txt")
assert not target.exists()
source.rename(target)
PY_FIX
python3 check_names.py sample
printf 'exit=%s\n' "$?"

Now expect {"collision_groups": 0, "groups": []} and exit=0. Checking symlink targets is a separate task; use the Python broken symlink audit for that.

5. Validation: exit codes and read-only behavior

After the previous renames, run this test. It adds fixtures only inside the demo directory; rebuild the entire demo directory before repeating it.

python3 - <<'PY_TEST'
import json
from pathlib import Path
import subprocess
import sys

root = Path("sample")
# Add a nested directory that must NOT be followed through a symlink.
outside = Path("outside")
outside.mkdir()
for name in ["A.txt", "a.txt"]:
    (outside / name).write_text("outside\n", encoding="utf-8")
(root / "external").symlink_to(outside.resolve(), target_is_directory=True)
(root / "missing-link").symlink_to("not-created")
(root / ".hidden").mkdir()
(root / ".hidden" / "ONLY.txt").write_text("hidden\n", encoding="utf-8")

def run(path):
    return subprocess.run(
        [sys.executable, "check_names.py", str(path)],
        capture_output=True, text=True, check=False,
    )

before = {str(p): p.read_bytes() for p in root.rglob("*")
          if not p.is_symlink() and p.is_file()}
result = run(root)
assert result.returncode == 0 and result.stderr == "", result
assert json.loads(result.stdout)["collision_groups"] == 0
print("PASS: clean tree; symlink targets not scanned")
(root / ".hidden" / "only.txt").write_text("second\n", encoding="utf-8")
result = run(root)
assert result.returncode == 1 and result.stderr == "", result
assert json.loads(result.stdout)["groups"] == [
    {"directory": ".hidden", "names": ["ONLY.txt", "only.txt"]}
]
print("PASS: hidden directory collision")
for bad in [root / "not-created", root / "notes.md"]:
    result = run(bad)
    assert result.returncode == 2 and not result.stdout
    assert result.stderr.startswith("ERROR:"), result
print("PASS: missing root and file root rejected")
assert all(Path(p).read_bytes() == data for p, data in before.items())
print("PASS: existing file contents unchanged")
PY_TEST

Expect four PASS lines. The test confirms hidden-directory traversal, skipping external directory symlink targets, failure for missing and regular-file roots, and unchanged existing file contents.

Exit codes: 0 means a complete scan with no candidate groups; 1 means a complete scan with findings; 2 means a scanning or CLI error. Scanning errors go to stderr with empty stdout. Gate later actions like this:

python3 check_names.py sample > filename-report.json && printf 'preflight passed\n'

The validation test leaves a hidden-directory collision, so this command currently returns 1 without printing the success message. Resolve the finding manually and rerun it. Redirection may create an empty report on error; callers must check the exit code too.

6. Troubleshooting: preflight is not automatic repair

7. Summary

Use NFC and casefold to flag candidate conflicts, then review original names and references before copying, backing up, or organizing static assets. Preserve the distinction between exit codes 0, 1, and 2 so an incomplete scan never looks clean. This preflight does not replace target-filesystem tests, content verification, or a Hugo build.