Fix CSV Encoding with Python: BOMs, GB18030, and UTF-16

A CSV opens on another machine but raises UnicodeDecodeError in a Python job on Linux. Or a column named sample_id arrives as \ufeffsample_id. Fixing the byte encoding before streaming CSV statistics keeps those problems separate from data validation.

This beginner-to-intermediate tutorial uses the Python standard library to inspect BOMs and convert known GB18030, BOM-marked UTF-16, or UTF-8 input into UTF-8 without a BOM. It changes encoding only, preserving fields, quoting, delimiters, and line endings.

1. Method: establish the source encoding

Follow this order:

1. Check the export tool’s encoding setting and retain the original file.
2. Inspect the initial bytes; use a BOM to guide the decoder choice.
3. If there is no BOM, confirm the encoding with the provider. Trying decoders until one succeeds does not establish correctness.

utf-8-sig decoding skips an optional leading UTF-8 BOM. No BOM does not establish GB18030, and successful decoding does not establish correct text. Avoid errors="ignore" and errors="replace": they can discard bytes or substitute characters. See Python’s encoding error handlers.

2. Setup: requirements and sample files

Use Python 3.11 or newer; no pip packages are required. These examples were tested on Linux with Python 3.11.4. Input is limited to 20 MiB. The script holds input bytes, decoded text, and output bytes in memory; it is not a large-file streaming converter.

Work in a new writable directory. These commands create files exclusively and fail rather than overwrite existing paths:

python3 - <<'PYTHON'
from pathlib import Path
text = 'sample_id,note\r\n001,中文\r\n002,"comma,inside"\r\n003,"line1\r\nline2"\r\n'
for name, encoding in (
    ("source-gb.csv", "gb18030"),
    ("source-bom.csv", "utf-8-sig"),
    ("source-utf16.csv", "utf-16"),
):
    with Path(name).open("xb") as handle:
        handle.write(text.encode(encoding))
PYTHON

Inspect the first four bytes:

python3 - <<'PYTHON'
from pathlib import Path
for name in ("source-gb.csv", "source-bom.csv", "source-utf16.csv"):
    with Path(name).open("rb") as handle:
        prefix = handle.read(4)
    print(name, prefix.hex(" "))
PYTHON

The GB18030 sample starts with the ASCII header bytes 73 61 6d 70; the UTF-8 BOM sample starts with ef bb bf 73. The UTF-16 BOM can be ff fe or fe ff, depending on the platform that created it. Removing those two bytes does not make the remaining content UTF-8.

3. Implementation: save the complete script

Save this as convert_csv_encoding.py:

"""Convert a small text/CSV file to UTF-8 without overwriting any file."""
import argparse
import codecs
from pathlib import Path
import sys

MAX_BYTES = 20 * 1024 * 1024


def bom_name(data):
    # UTF-32 must precede UTF-16: their prefixes can overlap.
    for mark, name in (
        (codecs.BOM_UTF32_LE, "utf-32-le"),
        (codecs.BOM_UTF32_BE, "utf-32-be"),
        (codecs.BOM_UTF8, "utf-8-sig"),
        (codecs.BOM_UTF16_LE, "utf-16-le"),
        (codecs.BOM_UTF16_BE, "utf-16-be"),
    ):
        if data.startswith(mark):
            return name
    return None


def convert(source, destination, encoding):
    with source.open("rb") as handle:
        data = handle.read(MAX_BYTES + 1)
    if len(data) > MAX_BYTES:
        raise ValueError("input exceeds the 20 MiB limit")
    bom = bom_name(data)
    allowed = {
        "utf-8-sig": {None, "utf-8-sig"},
        "gb18030": {None},
        "utf-16": {"utf-16-le", "utf-16-be"},
    }
    if bom not in allowed[encoding]:
        raise ValueError(f"BOM {bom!r} conflicts with {encoding}")
    text = data.decode(encoding, errors="strict")
    result = text.encode("utf-8", errors="strict")
    # Input validation finishes before exclusive output creation.
    with destination.open("xb") as handle:
        handle.write(result)
    return len(data), len(result)


def main():
    parser = argparse.ArgumentParser(description=__doc__, allow_abbrev=False)
    parser.add_argument("input", type=Path)
    parser.add_argument("output", type=Path)
    parser.add_argument("--from-encoding", required=True,
                        choices=("utf-8-sig", "gb18030", "utf-16"))
    args = parser.parse_args()
    try:
        before, after = convert(args.input, args.output, args.from_encoding)
    except (OSError, UnicodeError, ValueError) as exc:
        print(f"ERROR: {exc}", file=sys.stderr)
        return 1
    print(f"OK: input_bytes={before} output_bytes={after}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

The script reads a limited number of bytes, checks the BOM against the selected encoding, and strictly decodes and re-encodes the text. It checks UTF-32 before UTF-16 so ff fe 00 00 is not misidentified as UTF-16. UTF-32 is outside this example’s scope. UTF-16 input must have a BOM; for BOM-less input, establish byte order and re-export from the source tool first.

Output uses open() in x mode, refusing existing files, the input path itself, and existing symbolic links. A decoding failure occurs before output creation. Disk errors or interruption during writing can leave partial output. This is not an atomic commit; only a successful exit makes the result eligible for later processing.

Check usage, then run:

python3 --version
python3 convert_csv_encoding.py --help

python3 convert_csv_encoding.py source-gb.csv clean-gb.csv --from-encoding gb18030
python3 convert_csv_encoding.py source-bom.csv clean-bom.csv --from-encoding utf-8-sig
python3 convert_csv_encoding.py source-utf16.csv clean-utf16.csv --from-encoding utf-16

Success prints OK: input_bytes=... output_bytes=... and returns 0. Input, encoding, or file errors return 1; argparse returns 2 for invalid arguments. Conversion operates on the complete text without parsing or rewriting CSV, preserving CRLF, quoted commas, embedded line endings, and leading zeros in 001. It does not validate the CSV’s business rules.

4. Validation: check content as well as exit status

Run the same check against all three results:

python3 - <<'PYTHON'
import codecs
import csv
from pathlib import Path

expected = [
    ["sample_id", "note"],
    ["001", "中文"],
    ["002", "comma,inside"],
    ["003", "line1\r\nline2"],
]
outputs = [Path(name).read_bytes() for name in
           ("clean-gb.csv", "clean-bom.csv", "clean-utf16.csv")]
assert outputs[0] == outputs[1] == outputs[2]
assert not outputs[0].startswith(codecs.BOM_UTF8)
for name in ("clean-gb.csv", "clean-bom.csv", "clean-utf16.csv"):
    with open(name, encoding="utf-8", newline="") as handle:
        assert list(csv.reader(handle, strict=True)) == expected
print("PASS: identical UTF-8 bytes; IDs, quotes and embedded CRLF preserved")
PYTHON

Expected output:

PASS: identical UTF-8 bytes; IDs, quotes and embedded CRLF preserved

The CSV check uses newline="" to let the parser handle line endings, as described in Python’s CSV documentation. For real data, also inspect Chinese text, headers, record counts, and key IDs. Transcoding cannot recover text that was already saved as mojibake; obtain a fresh export in that case.

Check failure paths next. The first command deliberately reads GB18030 as UTF-8; the second attempts to overwrite an existing output:

python3 convert_csv_encoding.py source-gb.csv wrong.csv --from-encoding utf-8-sig
printf 'exit=%s\n' "$?"
python3 -c 'from pathlib import Path; assert not Path("wrong.csv").exists()'

python3 convert_csv_encoding.py source-bom.csv clean-bom.csv --from-encoding utf-8-sig
printf 'exit=%s\n' "$?"

Both should print ERROR: ... and exit=1. wrong.csv should not exist, and clean-bom.csv should remain unchanged. Rerun this section’s content check to confirm the overwrite attempt did not change it.

5. Automation and troubleshooting

Start the next step only after conversion succeeds. Use a new output name or a separate working directory for each run:

python3 convert_csv_encoding.py source-gb.csv ready.csv --from-encoding gb18030 &&
  python3 -c 'import csv; f=open("ready.csv", encoding="utf-8", newline=""); print(next(csv.reader(f))); f.close()'

The expected header is ['sample_id', 'note']. Follow with a duplicate ID audit if needed; encoding conversion does not establish uniqueness.

Keep the original file and a record of its encoding. A SHA-256 manifest can track original and converted files separately. Different encodings usually yield different checksums without implying that field values changed.

Summary

Establish the source encoding, then strictly decode and convert to UTF-8 instead of hiding lost content with error suppression. Keep the original, use a new output path, and check both exit status and actual CSV content. Proceed to counting, duplicate checks, or other automation only after those checks pass.