Read CSV Files Inside tar.gz with Python: Summaries Without Extraction

A daily export stores a CSV inside daily.tar.gz, but all you need is a small summary. Extracting the whole package adds disk usage and cleanup work. This beginner-to-intermediate tutorial uses Python’s standard library to read one CSV member directly, process records incrementally, and return a failing exit code when input checks fail.

1. Method: read an archive member as a file

Open the tar.gz, require one matching regular-file member, obtain its binary file object, wrap it as text, and feed it to the CSV parser. extractfile() returns a readable object; our script never calls the disk-writing extract() or extractall() methods. See the official tarfile documentation for the distinction.

Use trusted, completed local archives that remain unchanged during reading. The input must be UTF-8, optionally with a BOM, and have exactly the header id,value. An ID cannot be empty or all whitespace, and int() converts each value to an integer. A header-only file produces zero records; repeated IDs count as separate records.

Compared with streaming an ordinary CSV, the extra work is member selection and binary-to-text wrapping. We first scan with getmembers(), which stores member metadata and may spend time decompressing gzip data. CSV rows are processed individually rather than collected into a table. Use archives with a moderate member count; this is not a constant-memory guarantee for the entire archive operation.

2. Prepare the environment and demo archive

Use Python 3.9 or newer with working zlib support. No third-party packages are needed. The commands and script were tested on Linux with Python 3.11.4, using bash or zsh.

1. Run these commands in the same terminal. Pick a fresh directory name when repeating the experiment:

python3 --version
mkdir tar-csv-demo
cd tar-csv-demo
python3 - <<'PY_DATA'
import io
import tarfile
payload = b'\xef\xbb\xbfid,value\n001,10\n"sample,002",12\n003,20\n'
with tarfile.open("daily.tar.gz", "x:gz") as archive:
    info = tarfile.TarInfo("reports/daily.csv")
    info.size = len(payload)
    archive.addfile(info, io.BytesIO(payload))
PY_DATA
python3 -m tarfile -l daily.tar.gz

The listing should include reports/daily.csv. The fixture constructs CSV bytes in memory without creating a reports/ directory. Mode x:gz refuses to overwrite an existing archive.

2. Save the next section as read_tar_csv.py, or download the complete script. The member argument must match the listing exactly: ./reports/daily.csv and reports/daily.csv are different strings here.

3. Complete implementation

"""Summarize one CSV member in a completed, trusted tar.gz. Python 3.9+."""
import argparse
import csv
import io
import json
import sys
import tarfile
import zlib


def summarize(path, member, max_bytes):
    with tarfile.open(path, "r:gz") as archive:
        matches = [item for item in archive.getmembers() if item.name == member]
        if not matches:
            raise ValueError(f"member not found: {member!r}")
        if len(matches) != 1:
            raise ValueError(f"duplicate member: {member!r}")
        info = matches[0]
        if not info.isfile():
            raise ValueError(f"member is not a regular file: {member!r}")
        if info.size > max_bytes:
            raise ValueError(f"member size {info.size} exceeds limit {max_bytes}")
        records = total = 0
        with archive.extractfile(info) as raw:
            with io.TextIOWrapper(raw, encoding="utf-8-sig", newline="") as text:
                reader = csv.reader(text, strict=True)
                if next(reader, None) != ["id", "value"]:
                    raise ValueError("expected header: id,value")
                for row in reader:
                    where = f"physical line {reader.line_num}"
                    if len(row) != 2 or not row[0].strip():
                        raise ValueError(f"{where}: expected nonempty id and value")
                    try:
                        value = int(row[1])
                    except ValueError as exc:
                        raise ValueError(f"{where}: value must be an integer") from exc
                    records += 1
                    total += value
    return {"member": member, "records": records, "total": total}


def main():
    parser = argparse.ArgumentParser(description=__doc__, allow_abbrev=False)
    parser.add_argument("archive", help="completed local tar.gz")
    parser.add_argument("member", help="exact member name, as listed by tarfile")
    parser.add_argument("--max-bytes", type=int, default=64 * 1024 * 1024,
                        help="maximum declared CSV size; default: 67108864")
    args = parser.parse_args()
    if args.max_bytes <= 0:
        parser.error("--max-bytes must be positive")
    try:
        result = summarize(args.archive, args.member, args.max_bytes)
    except (OSError, EOFError, tarfile.TarError, zlib.error,
            UnicodeError, csv.Error, ValueError) as exc:
        print(f"ERROR: {exc}", file=sys.stderr)
        return 1
    print(json.dumps(result, ensure_ascii=False))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

utf-8-sig accepts UTF-8 with or without a BOM and fails strictly on other encodings. The TextIOWrapper documentation describes wrapping binary streams as text. As the CSV documentation specifies, newline="" lets the parser handle newlines inside quoted fields.

The target member’s declared size is limited to 64 MiB by default; adjust it with --max-bytes. This does not limit the entire archive’s decompressed size, member count, or execution time, and does not isolate hostile archives. The CSV parser also retains its standard field-size limit; unusually large fields fail.

4. Run and validate

1. Run the help command and normal example:

python3 read_tar_csv.py --help
python3 read_tar_csv.py daily.tar.gz reports/daily.csv

{"member": "reports/daily.csv", "records": 3, "total": 42}

"sample,002" is one ID, so splitting lines with split(",") would be wrong. Success writes only JSON to stdout and exits 0. Read or data-check failures write ERROR only to stderr and exit 1. Invalid arguments return argparse’s exit code 2.

2. Test missing members and the size limit:

python3 read_tar_csv.py daily.tar.gz missing.csv
printf 'exit=%s\n' "$?"
python3 read_tar_csv.py daily.tar.gz reports/daily.csv --max-bytes 10
printf 'exit=%s\n' "$?"

Both commands should print ERROR and exit=1, with no success JSON.

3. Verify that duplicate members are rejected:

python3 - <<'PY_DUP'
import io
import tarfile
payload = b"id,value\n001,1\n"
with tarfile.open("duplicate.tar.gz", "x:gz") as archive:
    for _ in range(2):
        info = tarfile.TarInfo("reports/daily.csv")
        info.size = len(payload)
        archive.addfile(info, io.BytesIO(payload))
PY_DUP
python3 read_tar_csv.py duplicate.tar.gz reports/daily.csv
printf 'exit=%s\n' "$?"

Expect duplicate member and exit code 1. Scanning members before reading detects the duplicate before any statistics are computed.

4. In the fresh demo directory, confirm that no CSV was extracted:

python3 - <<'PY_CHECK'
from pathlib import Path
assert not Path("reports").exists()
assert not Path("daily.csv").exists()
print("OK: no extracted CSV")
PY_CHECK

The script validates reading and formatting of the selected CSV, not the integrity of the whole package: the end of a tar archive does not necessarily coincide with the end of its gzip byte stream. For backup acceptance, separately compare a SHA-256 file manifest.

5. Troubleshooting

For source-file header rules, see the CSV schema validation tutorial, then rebuild the archive after fixing it.

6. Summary

Connect extractfile() to a text wrapper and CSV parser to summarize a tar.gz member without extraction or cleanup. Exact member names, duplicate rejection, header checks, and failing exit codes make the script useful in daily automation. Validate the data rules with a small archive before adjusting the target-member size limit for real exports.