Stream Large CSV Files with Python: Validate Rows and Count Labels

A CSV file can be too large to load into a list even when the question is simple: how many records have each label? You can answer that question in one pass while keeping only the current record and a few counters.

This tutorial builds a small command-line tool using Python’s standard library. It reads a sample-label table, checks every record, and prints a JSON summary only after the entire input passes. It is a useful first check before joining protein sequences and labels.

1. Set the input rules

The example accepts a UTF-8 CSV, with or without a byte-order mark, containing exactly these columns in this order:

Here, unknown means the label is unavailable. Keep it separate from a negative label. The values and records below are invented for the tutorial.

The tool counts records, not unique samples. Repeated IDs are counted again; checking uniqueness requires a separate policy and additional state. IDs stay as strings, so 001 is not converted to 1. A header-only file is valid and produces zero counts. A completely empty file fails because it has no header.

2. Download and run

Use Python 3.11 or newer. No third-party packages are needed. The example was tested with Python 3.14.7.

Download summarize_labels.py, save it in a new directory, then create this input and run the script:

cat > labels.csv <<'CSV'
sample_id,label
001,1
002,0
"sample,003",unknown
004,1
005,0
006,1
CSV
python3 summarize_labels.py --help
python3 summarize_labels.py labels.csv

Expected output:

{
  "records": 6,
  "label_counts": {
    "0": 2,
    "1": 3,
    "unknown": 1
  }
}

The comma inside "sample,003" belongs to the ID. A CSV parser handles that quoted field; splitting each line with line.split(",") would produce the wrong number of fields. Quoted fields can also contain line breaks, so a physical line is not necessarily a complete record. See the Python CSV documentation.

3. The complete script

"""Stream a UTF-8 CSV with exactly sample_id,label columns; count 0/1/unknown."""
import argparse
import csv
import json


def summarize(path):
    counts = {"0": 0, "1": 0, "unknown": 0}
    records = 0
    with open(path, encoding="utf-8-sig", newline="") as handle:
        reader = csv.reader(handle, strict=True)
        try:
            header = next(reader, None)
            if header != ["sample_id", "label"]:
                raise ValueError("Expected header: sample_id,label (in that order)")
            for row in reader:
                where = f"Record ending at physical line {reader.line_num}"
                if len(row) != 2:
                    raise ValueError(f"{where}: expected 2 fields, got {len(row)}")
                sample_id, label = row
                if not sample_id or sample_id != sample_id.strip():
                    raise ValueError(f"{where}: sample_id is empty or has outer whitespace")
                if label not in counts:
                    raise ValueError(f"{where}: label must be 0, 1, or unknown")
                counts[label] += 1
                records += 1
        except csv.Error as exc:
            raise ValueError(f"CSV parse error near physical line {reader.line_num}: {exc}") from exc
    return {"records": records, "label_counts": counts}


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("input", help="Path to the input CSV")
    args = parser.parse_args()
    try:
        result = summarize(args.input)
    except (OSError, UnicodeError, ValueError) as exc:
        parser.exit(1, f"error: {exc}\n")
    print(json.dumps(result, indent=2))


if __name__ == "__main__":
    main()

The file is opened with newline="" so the CSV parser handles newline conventions. utf-8-sig accepts UTF-8 input with an optional initial BOM. The parser returns strings, and strict=True makes it raise an error for bad CSV input it detects. These parser settings do not replace the explicit checks for our header, field count, IDs, and allowed labels. Python documents the CSV reader and its dialect options here.

The report is printed after the file closes successfully. If record 500,000 fails, the process exits with status 1, sends a diagnostic to standard error, and prints no summary to standard output. Earlier records have been counted internally, but those incomplete counts are not presented as a finished report.

4. Check failure behavior

An unexpected label should fail visibly:

cat > bad-labels.csv <<'CSV'
sample_id,label
001,1
002,maybe
CSV
python3 summarize_labels.py bad-labels.csv
echo "exit=$?"

Expected diagnostic and exit status:

error: Record ending at physical line 3: label must be 0, 1, or unknown
exit=1

The physical line number helps locate a problem even when quoted records span several lines. The input file is opened only for reading.

If you save the summary with shell redirection, remember that the shell can create or truncate the destination before Python validates the input. In an automated job, write to a temporary path, check the exit status, and only then replace the last successful report.

5. What streaming saves—and what it does not

This script does not retain earlier records. Its working data is roughly one parsed record, input buffers, and three counters, rather than a list that grows with the number of rows. Counter integers grow as counts increase, and unusually large individual records still require memory. This is not a hard memory limit.

The fixed label set matters. Counting arbitrary sample IDs in a dictionary would require one entry per distinct ID. Reading the input as a stream would not prevent that dictionary from growing. Likewise, converting the reader to list(reader) would load the remaining records into memory.

Streaming also does not remove the cost of reading and parsing the entire file. It is a good fit for totals and other small summaries. Global sorting, exact deduplication, and large joins need a different storage strategy once their working data no longer fits in memory.

The local checks covered the six-record example, a 100,000-record input with equal numbers of 0 and 1, BOM and CRLF handling, quoted commas and newlines, header-only input, repeated IDs, and malformed records. These checks establish behavior for those cases; they are not a speed benchmark.

6. Troubleshooting

For reproducible runs, keep the source file unchanged while reading it and record its checksum with the dataset manifest workflow. This tool does not lock the file or take a snapshot.

7. Keep the summary small

Read one CSV record at a time, validate it, and keep only the state your result needs. For this label audit, three counters are enough; checking ID uniqueness or joining datasets is a separate task.