Stream XML to CSV with Python iterparse: Release Nodes and Preserve Complete Reports
On this page
A growing XML export can exhaust memory when ET.parse() builds the entire tree. Switching to iterparse() is not enough if the root still retains every completed record.
This beginner-to-intermediate tutorial uses the Python standard library to stream a local inventory XML file into CSV, preserving the previous report if the input is broken at the end. It accepts only trusted, known-structure, ordinary local files that stay unchanged during conversion, not uploads or untrusted network XML.
1. Method: process complete records on end events
- Require one
inventoryroot whose direct children areitemrecords. Each record needs a nonemptyskuattribute and exactly one plain-textnameandquantitychild; quantities must be ASCII nonnegative integer strings. - Read fields on the item’s
endevent. After writing a row, callroot.remove(node)and clear the node so completed siblings do not accumulate. - Write CSV to a temporary file in the output directory. Replace the final report only after the entire input parses and both files close. Failure returns 1 and leaves the old report in place; success overwrites an existing report.
The official ElementTree reference describes blocking reads and recommends end events for fully populated elements. clear() alone empties an element but leaves it attached to its parent, so we explicitly remove completed direct children. Retained data mainly follows individual record size and parser buffering instead of a list of all records. This is not a fixed memory ceiling: a huge individual element still consumes memory.
For URL checks on a Hugo sitemap, use the existing Hugo sitemap checker. This tutorial handles streaming conversion of repeated XML records, not sitemap-specific validation rules.
2. Setup: create an isolated demo directory
Use an existing Python 3.9 or newer installation; no third-party packages are required. Commands were tested on Linux with Python 3.11.4 and work in Bash or zsh.
- Create a new directory in the same terminal. Use a different name for repeat experiments to avoid overwriting fixtures:
python3 --version
mkdir xml-stream-demo
cd xml-stream-demo
cat > inventory.xml <<'XML'
<?xml version="1.0" encoding="UTF-8"?>
<inventory xmlns="urn:demo:inventory">
<item sku="001"><name>螺母, M6</name><quantity>12</quantity></item>
<item sku="002"><name>Washer & spacer</name><quantity>0</quantity></item>
</inventory>
XML
- Save the next section as
xml_to_csv.py. Run subsequent commands in this directory. Your real output directory must already exist and be under your control.
3. Implementation: stream records, then replace the report
"""Stream trusted local inventory XML into CSV. Python 3.9+."""
import argparse
import csv
import os
from pathlib import Path
import sys
import tempfile
import xml.etree.ElementTree as ET
NS = "{urn:demo:inventory}"
def field(item, name):
matches = item.findall(NS + name)
if len(matches) != 1 or len(matches[0]) != 0:
raise ValueError(f"expected one plain-text {name}")
value = (matches[0].text or "").strip()
if not value:
raise ValueError(f"empty {name}")
return value
def convert(source, target):
writer = csv.writer(target, lineterminator="\n")
writer.writerow(["sku", "name", "quantity"])
root = None
depth = count = 0
for event, node in ET.iterparse(source, events=("start", "end")):
if event == "start":
depth += 1
if depth == 1:
root = node
if root.tag != NS + "inventory":
raise ValueError("expected inventory root with demo namespace")
continue
if depth == 2:
if node.tag != NS + "item":
raise ValueError("expected item directly under inventory")
sku = node.get("sku", "").strip()
if not sku:
raise ValueError("missing sku attribute")
name = field(node, "name")
quantity = field(node, "quantity")
if not quantity.isascii() or not quantity.isdecimal():
raise ValueError("quantity must contain ASCII digits only")
writer.writerow([sku, name, int(quantity)])
count += 1
root.remove(node)
node.clear()
depth -= 1
if count == 0:
raise ValueError("inventory contains no items")
return count
def main():
parser = argparse.ArgumentParser(description=__doc__, allow_abbrev=False)
parser.add_argument("input", type=Path)
parser.add_argument("output", type=Path)
args = parser.parse_args()
temporary = None
try:
if args.input.resolve() == args.output.resolve():
raise ValueError("input and output must be different paths")
with args.input.open("rb") as source:
with tempfile.NamedTemporaryFile(
mode="w", encoding="utf-8", newline="",
dir=args.output.parent, prefix=".xml-csv-", delete=False,
) as target:
temporary = Path(target.name)
count = convert(source, target)
os.replace(temporary, args.output)
except (OSError, ValueError, ET.ParseError) as exc:
print(f"ERROR: {exc}", file=sys.stderr)
return 1
finally:
if temporary is not None:
temporary.unlink(missing_ok=True)
print(f"OK: {count} items")
return 0
if __name__ == "__main__":
raise SystemExit(main())
Use the expanded namespace name {urn:demo:inventory}, not a bare name lookup. A default namespace and a different prefix work identically when their URI matches; sku is an unqualified attribute. Field checks cover only the fields required here. Extra children inside item are ignored; this is neither full XML Schema validation nor a global SKU uniqueness check.
The temporary file shares the destination directory, avoiding cross-filesystem replacement. Ordinary parse and field errors close and remove it. This protects publication of complete results, not durability after power loss. Replacement uses the temporary file’s permissions rather than preserving old report permissions; see atomic JSON writes with Python for the related writing pattern.
4. Usage: help, output, and exit status
python3 xml_to_csv.py --help
python3 xml_to_csv.py inventory.xml inventory.csv
Successful output:
OK: 2 items
The resulting inventory.csv quotes the comma-containing field and decodes XML’s &:
sku,name,quantity
001,"螺母, M6",12
002,Washer & spacer,0
File or data errors return 1 and write an error to stderr; command-line usage errors return 2. Success returns 0 and prints only a count to stdout, keeping CSV separate from logs. Do not check only whether the destination exists: on failure it may still be yesterday’s report. Always check the exit status.
5. Validation: broken endings and 100,000 records
- Read the CSV back and deliberately remove the root closing tag. Even though rows have already reached the temporary file, this late error must not replace the old report:
python3 - <<'PY_CHECK'
import csv
from pathlib import Path
import subprocess
import sys
with open("inventory.csv", encoding="utf-8", newline="") as source:
assert list(csv.reader(source)) == [
["sku", "name", "quantity"],
["001", "螺母, M6", "12"],
["002", "Washer & spacer", "0"],
]
before = Path("inventory.csv").read_bytes()
original = Path("inventory.xml").read_bytes()
Path("truncated.xml").write_bytes(original.replace(b"</inventory>", b""))
result = subprocess.run(
[sys.executable, "xml_to_csv.py", "truncated.xml", "inventory.csv"],
capture_output=True, text=True,
)
assert result.returncode == 1, result
assert result.stdout == "" and result.stderr.startswith("ERROR:")
assert Path("inventory.csv").read_bytes() == before
assert Path("inventory.xml").read_bytes() == original
assert not list(Path(".").glob(".xml-csv-*"))
print("PASS: CSV round-trip; truncated XML preserves old output")
PY_CHECK
- Generate 100,000 small records, convert them, and count CSV records without collecting the file into a list:
python3 - <<'PY_DATA'
with open("many.xml", "w", encoding="utf-8") as target:
target.write('<inventory xmlns="urn:demo:inventory">\n')
for index in range(100000):
target.write(
f'<item sku="{index:06d}"><name>Part {index}</name>'
'<quantity>1</quantity></item>\n'
)
target.write('</inventory>\n')
PY_DATA
python3 xml_to_csv.py many.xml many.csv
python3 - <<'PY_COUNT'
import csv
with open("many.csv", encoding="utf-8", newline="") as source:
rows = csv.reader(source)
assert next(rows) == ["sku", "name", "quantity"]
assert sum(1 for _ in rows) == 100000
print("PASS: 100000 CSV records")
PY_COUNT
Expect OK: 100000 items and PASS: 100000 CSV records. This validates conversion and record counts, not a cross-machine memory benchmark. Peak-memory comparisons need fixed Python versions, input shapes, and measurement tools. Keep downstream CSV work streaming record by record too.
6. Troubleshooting: causes and direct fixes
| Problem | Cause and fix |
|---|---|
| Root namespace error | Check the actual root tag and URI; do not strip all namespaces to force a match. |
expected one plain-text name |
A field is missing, duplicated, or contains child markup. Fix the input or explicitly design mixed-content handling. |
| Invalid quantity | Blank values, signs, decimals, and non-ASCII digits violate this example’s contract. Correct the data instead of silently using zero. |
| Memory keeps growing | Do not collect records into a list or use only clear(). Confirm that input records really are direct children of the root. |
| Output directory missing or unwritable | Choose an existing writable directory; do not broadly relax system permissions. |
ParseError at the end |
Repair or re-export the source before rerunning. Keeping the old report does not make the new run successful. |
The official Python XML security guidance warns that malicious XML can exhaust resources. Streaming is not a security sandbox; do not expose this script directly to untrusted uploads. It also does not neutralize spreadsheet formulas. Apply a separate receiver-specific policy before importing external text into spreadsheet software.
7. Summary
For large sets of repeated XML records, read complete records on end events and remove completed siblings instead of retaining the entire tree. Writing temporary CSV and replacing the report only after full success preserves old results when the input is truncated. Callers must still check exit status and adapt namespace and field rules to their actual XML structure.
- Author: yanzhi
- Original article: https://www.bobobk.com/en/python-stream-xml-iterparse-csv.html
- License: This work is licensed under CC BY-NC-ND 4.0. See the license for attribution, noncommercial use and no-derivatives terms.
Related articles
- Diagnose CSV Delimiters with Python: Sniffer, Explicit Formats, and Full Validation
- Merge CSV Shards in Numeric Order with Python: Headers and Failure-Safe Output
- Fix Python CSV Field Limits: Set a Character Cap and Test Boundaries
- Compare CSV Snapshots by ID with Python: Added, Removed, and Changed Records
- Split Large CSV Files with Python: Headers, Multiline Fields, and Completion Checks