Sample a Large CSV with Python: Reservoir Sampling with a Fixed Seed

You need a small random sample from a CSV that is too large to load into memory. Taking the first thousand rows can overrepresent whichever source, label, or date happens to come first. Reservoir sampling selects a fixed-size sample while reading the file once.

This tutorial builds a standard-library Python command that preserves the header, samples complete CSV records, and writes selected records in their original order. It complements the earlier guide to streaming and validating large CSV files. The examples and seven tests were run with Python 3.14.7; no large-file performance benchmark is claimed.

1. Define what is being sampled

The sampling unit is a data record, excluding the header. For N records and a requested size k, the output contains min(k, N) records. When N is at least k, each record has inclusion probability k/N under uniform random draws. The script uses the basic reservoir method known as Algorithm R, described in Vitter’s paper on random sampling.

The script preserves field values as strings, so 001 remains 001. It checks table shape, not the scientific meaning of labels or sequences. A truly blank record fails the field-count check. Empty cells are otherwise allowed.

2. Download and try a small example

Use Python 3.11 or newer; no third-party packages are needed. Start in a new directory and download sample_csv.py:

mkdir csv-sampling-demo
cd csv-sampling-demo
curl -fL https://www.bobobk.com/downloads/reservoir/sample_csv.py \
    -o sample_csv.py
python3 sample_csv.py --help

Create this invented six-record table and sample three records:

cat > input.csv <<'CSV'
sample_id,label
001,1
002,0
003,1
004,0
005,unknown
006,1
CSV
python3 sample_csv.py input.csv sample.csv --size 3 --seed 42

The command prints this JSON summary:

{
  "input_records": 6,
  "requested_records": 3,
  "sampled_records": 3,
  "seed": 42
}

With the tested Python version, sample.csv contains:

sample_id,label
002,0
003,1
005,unknown

The output CSV is rewritten with standard CSV quoting and record terminators. Its field values are preserved; byte-for-byte formatting of the original file is not.

3. The complete sampler

Save this as sample_csv.py if you prefer to inspect and copy the implementation:

#!/usr/bin/env python3
"""Sample CSV records in one pass, then write a new output file."""
import argparse
import csv
import json
from pathlib import Path
import random


def sample_csv(source, size, seed):
    if size < 1:
        raise ValueError("sample size must be positive")
    rng = random.Random(seed)
    reservoir = []
    seen = 0
    with source.open(encoding="utf-8-sig", newline="") as handle:
        reader = csv.reader(handle, strict=True)
        header = next(reader, None)
        if not header or any(not name.strip() for name in header):
            raise ValueError("a header with nonempty column names is required")
        if len(set(header)) != len(header):
            raise ValueError("duplicate column names")
        for seen, row in enumerate(reader, start=1):
            if len(row) != len(header):
                raise ValueError(
                    f"record {seen}: expected {len(header)} fields, got {len(row)}"
                )
            item = (seen, row)
            if seen <= size:
                reservoir.append(item)
            else:
                slot = rng.randrange(seen)
                if slot < size:
                    reservoir[slot] = item
    reservoir.sort(key=lambda item: item[0])
    return header, [row for _, row in reservoir], seen


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("input", type=Path)
    parser.add_argument("output", type=Path)
    parser.add_argument("--size", required=True, type=int)
    parser.add_argument("--seed", type=int, default=42)
    args = parser.parse_args()
    if args.size < 1:
        parser.error("--size must be positive")
    try:
        header, rows, total = sample_csv(args.input, args.size, args.seed)
        # Exclusive creation refuses to overwrite any existing destination.
        with args.output.open("x", encoding="utf-8", newline="") as handle:
            writer = csv.writer(handle)
            writer.writerow(header)
            writer.writerows(rows)
    except (OSError, UnicodeError, csv.Error, ValueError) as exc:
        parser.exit(1, f"ERROR: {exc}\n")
    print(json.dumps({
        "input_records": total,
        "requested_records": args.size,
        "sampled_records": len(rows),
        "seed": args.seed,
    }, indent=2))


if __name__ == "__main__":
    main()

csv.reader handles record boundaries, including newlines inside quoted fields. Files are opened with newline="", as recommended by the Python CSV documentation. Splitting on commas or sampling physical lines would break these cases.

The input is read and checked before the output is opened. Thus a parsing or shape error does not create the destination. Exclusive mode "x" prevents replacement of an existing file, including the input itself. A write failure or interrupted process can still leave a partial new output; use only a file from a command that completed successfully.

4. Why later rows still have a chance

The first k records fill the reservoir. For record number i after that, draw an integer from 0 through i−1. If it is below k, replace that reservoir slot; otherwise keep the current sample.

The new record enters with probability k/i. A previously selected record survives the step with probability 1−1/i. Combined with its earlier inclusion probability k/(i−1), its new inclusion probability is also k/i. The final sort only restores source order; it does not change which records were selected.

The code scans all N records and retains at most k sampled records, plus the current record and parser buffers. Memory therefore depends on the sampled rows’ sizes, not just their count. For bounded-width rows, the scan is O(N), final sorting is O(k log k), and retained sample storage is O(k). Selecting a small k reduces memory use, but it does not avoid reading the entire input.

5. Check repeatability before using a larger file

Run the same input again under a new output name:

python3 sample_csv.py input.csv sample-repeat.csv --size 3 --seed 42
python3 - <<'PY'
import csv
from pathlib import Path

assert Path("sample.csv").read_bytes() == Path("sample-repeat.csv").read_bytes()
with open("sample.csv", encoding="utf-8", newline="") as handle:
    rows = list(csv.reader(handle))
assert rows[0] == ["sample_id", "label"]
assert len(rows) - 1 == 3
print("PASS: repeatable sample, header and record count")
PY

The final line should be:

PASS: repeatable sample, header and record count

For your own existing large.csv, choose a new destination:

python3 sample_csv.py large.csv sample-1000.csv --size 1000 --seed 42
python3 --version

A fixed seed helps reproduce a run when input content, input order, code, and Python environment are held constant. The script uses a private random.Random instance. Do not assume randrange will produce an identical sample across every future Python version; the random module’s reproducibility notes distinguish stable guarantees from algorithms that may change.

Record the seed, requested size, script version, and Python version. Use a SHA-256 manifest to identify the input, and retain the resulting sample. Reordering the same rows can change which records are selected. Changing the seed also does not guarantee a different sample on a small dataset.

6. Common errors and limits

To see the failure behavior, create a row with a missing field:

cat > bad.csv <<'CSV'
sample_id,label
001,1
002
CSV
python3 sample_csv.py bad.csv bad-sample.csv --size 1 --seed 42

It exits with a nonzero status and reports:

ERROR: record 2: expected 2 fields, got 1

The record number excludes the header and counts logical CSV records, not physical lines. Even with --size 1, the malformed second record is checked.

Uniform row sampling does not balance labels. If one ID appears ten times, its ten records get ten opportunities to be sampled; this is not sampling unique entities. Audit duplicate CSV IDs first if IDs are your intended sampling units. For model evaluation, use a group-aware split when related sequences or repeated measurements must stay together.

7. Summary

Reservoir sampling produces a small sample without holding the whole CSV in memory. Parse complete records, keep the seed and input identity, and validate the full stream before using the output. Use a separate sampling design when you need label balance, unique entities, or separation between related groups.