Verify Research Datasets with SHA-256 Manifests in Python

A fixed random seed cannot reproduce an experiment if the input files have changed. A label correction, a regenerated split table, or a replaced FASTA file can leave the filename unchanged while changing the experiment.

This tutorial builds a small Python command that records a directory’s file paths and SHA-256 digests, then checks them before the next run. It fits after the FASTA audit and grouped split workflow: validate the data, freeze the assignment, and record the exact files used.

1. Decide what the manifest covers

Put the inputs needed for one experiment in a dedicated directory. Keep model outputs, caches, logs, and the manifest itself outside it.

experiment/
  inputs/
    proteins.fasta
    labels.csv
    split.csv
  inputs-v1.json
  dataset_manifest.py

Our manifest maps each relative file path to its digest. Moving the entire input directory does not change that mapping. Renaming a file does: the old name becomes missing and the new name becomes added.

This is a byte-level check. Rewrapping a FASTA sequence or converting line endings will change its file digest even if a parser produces the same sequences. Keep the raw-file digest for provenance; if you also need sequence-level deduplication, save a separate canonical-sequence identifier using a documented normalization policy.

The script includes hidden files, rejects symbolic links and special files, and ignores empty directories. It records neither permissions nor modification times. Use it on a local directory that is no longer being written; the scan is not an atomic filesystem snapshot.

2. Download and run the example

Use Python 3.11 or newer; no third-party packages are needed. This example was tested with Python 3.14.7 on macOS. The hashing call uses hashlib.file_digest, available since Python 3.11, with a file opened in binary mode.

Download dataset_manifest.py, or copy the complete script below. Run these commands in a new working directory after saving the script there. The shell examples use Bash or Zsh.

python3 --version
python3 dataset_manifest.py --help
mkdir inputs
python3 - <<'PY'
from pathlib import Path
Path("inputs/proteins.fasta").write_bytes(b">p1\nACDE\n")
Path("inputs/labels.csv").write_bytes(b"sample_id,label\np1,1\n")
Path("inputs/split.csv").write_bytes(b"sample_id,split\np1,train\n")
PY
python3 dataset_manifest.py create inputs inputs-v1.json
python3 dataset_manifest.py verify inputs inputs-v1.json

Expected output from the last two commands:

CREATED: 3 files
OK: 3 files match

The three files are invented test inputs, not a usable training dataset. The script creates the manifest exclusively: an existing inputs-v1.json causes an error instead of being overwritten. Keep that original manifest when investigating an unexpected change.

3. The complete script

"""Byte-level manifest for a quiet local dataset directory. Python 3.11+."""
import argparse
import hashlib
import json
import os
from pathlib import Path
import re
import stat
import sys


def scan(root):
    def fail(error):
        raise error

    files = {}
    for folder, dirs, names in os.walk(root, onerror=fail, followlinks=False):
        for name in dirs + names:
            path = Path(folder) / name
            mode = path.lstat().st_mode
            if stat.S_ISLNK(mode):
                raise ValueError(f"Symlink is not supported: {path}")
            if stat.S_ISDIR(mode):
                continue
            if not stat.S_ISREG(mode):
                raise ValueError(f"Not a regular file: {path}")
            with path.open("rb") as handle:
                digest = hashlib.file_digest(handle, "sha256").hexdigest()
            files[path.relative_to(root).as_posix()] = digest
    return files


def read_manifest(path):
    data = json.loads(path.read_text(encoding="utf-8"))
    if (not isinstance(data, dict) or data.get("version") != 1
            or data.get("algorithm") != "sha256"
            or not isinstance(data.get("files"), dict)):
        raise ValueError("Unsupported manifest format")
    for name, digest in data["files"].items():
        if (not name or not isinstance(digest, str)
                or re.fullmatch(r"[0-9a-f]{64}", digest) is None):
            raise ValueError("Invalid manifest entry")
    return data["files"]


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("action", choices=["create", "verify"])
    parser.add_argument("root", type=Path)
    parser.add_argument("manifest", type=Path)
    args = parser.parse_args()
    try:
        if args.root.is_symlink():
            raise ValueError("Dataset root must not be a symlink")
        root = args.root.resolve(strict=True)
        if not root.is_dir():
            raise ValueError("Dataset root must be a directory")
        manifest = args.manifest.resolve()
        if manifest.is_relative_to(root):
            raise ValueError("Keep the manifest outside the dataset directory")
        expected = read_manifest(manifest) if args.action == "verify" else None
        current = scan(root)
        if args.action == "create":
            data = {"version": 1, "algorithm": "sha256", "files": current}
            with manifest.open("x", encoding="utf-8", newline="\n") as handle:
                handle.write(json.dumps(data, sort_keys=True, indent=2) + "\n")
            print(f"CREATED: {len(current)} files")
            return 0
        missing = sorted(expected.keys() - current.keys())
        added = sorted(current.keys() - expected.keys())
        changed = sorted(name for name in expected.keys() & current.keys()
                         if expected[name] != current[name])
        for label, names in [("MISSING", missing), ("ADDED", added),
                             ("CHANGED", changed)]:
            for name in names:
                print(f"{label}: {json.dumps(name, ensure_ascii=False)}")
        if missing or added or changed:
            return 1
        print(f"OK: {len(current)} files match")
        return 0
    except (OSError, ValueError) as error:
        print(f"ERROR: {error}", file=sys.stderr)
        return 2


if __name__ == "__main__":
    sys.exit(main())

The directory scan uses os.walk with an error handler that raises traversal errors. A folder that cannot be read must not silently disappear from the inventory. Each discovered entry is checked before hashing, including symlinked directories.

The output uses sorted JSON keys and relative paths with / separators. This makes manifests easier to compare in version control. During verification, paths come from a fresh directory scan; the program compares manifest keys rather than opening paths supplied by the JSON file.

4. Confirm that changes are detected

Change one residue in the demonstration file while keeping its byte count unchanged:

python3 - <<'PY'
from pathlib import Path
Path("inputs/proteins.fasta").write_bytes(b">p1\nACDF\n")
PY
python3 dataset_manifest.py verify inputs inputs-v1.json
printf 'exit code: %s\n' "$?"

Expected output:

CHANGED: "proteins.fasta"
exit code: 1

That catches a change a size-only check would miss. Adding a file produces ADDED; deleting a tracked file produces MISSING. A single verification can report all three categories. Names are JSON-quoted so unusual characters such as embedded newlines remain distinguishable in the report.

For a pipeline, make successful verification a prerequisite for the next command. After reviewing changes and establishing the intended input version, a small wrapper can stop on failure:

#!/usr/bin/env bash
set -euo pipefail
python3 dataset_manifest.py verify inputs inputs-v1.json
# Place the real training or analysis command after the verification line.

This only checks the directory at verification time. Keep it unchanged while the analysis reads it, or run against an immutable snapshot.

5. Attach the manifest to the experiment

A manifest is most useful when it travels with the result. Save it alongside the code revision, environment lockfile, model identifier or checkpoint digest, training configuration, and seed values. For protein model comparisons, include the exact split table and label file in the hashed directory, not just the FASTA file.

If a correction is intentional, document it and create inputs-v2.json. Keep inputs-v1.json with earlier results. Regenerating a manifest immediately after a failed check erases the comparison you were trying to make.

A checksum does not validate labels, detect homologous leakage, or guarantee identical model output. Nor does a local JSON file establish who supplied the data: someone who can replace both the data and manifest can create a matching pair. Keep the reference manifest in a trusted, versioned location. This script is for accidental-change detection in a controlled research workflow, not hostile concurrent filesystem access.

6. Troubleshooting

The downloadable script was checked for successful verification, a same-length content change, added and missing files, relocation, deterministic output, existing-manifest protection, malformed input, symlinks, special files, and empty directories. These checks validate the example’s file-handling behavior; they do not validate a biological dataset.

Keep an exact input record

Freeze the input directory, create its manifest, and save that manifest with the experiment. Verify it before reusing the inputs, and treat a mismatch as something to investigate. Together with the split assignment and software environment, it gives later comparisons a concrete record of which files were used.

The cover is conceptual artwork, not an experimental protein structure. Python documentation checked September 27, 2026.