Check Hugo Sitemaps with Python: Namespaces, Duplicate URLs, and Required Pages
On this page
Hugo builds successfully, yet a new post is absent from the sitemap. Two source files may also share a permalink and produce duplicate URLs. Checking that the XML file exists will miss both problems.
This tutorial is for beginner-to-intermediate readers with an existing Hugo project. Use the Python standard library to check local sitemap namespaces, duplicate URLs, and required pages. Start with a two-URL fixture, then check a temporary build; pair it with the internal link checker to inspect links and images within articles.
1. Method: distinguish page lists from indexes
The official Hugo sitemap documentation describes a page sitemap for a monolingual project and an additional root sitemap index for a multilingual project. A <urlset> lists pages; a <sitemapindex> lists child sitemap locations. This script accepts only the former, so supply child files explicitly.
The Sitemaps protocol requires its namespace and a <loc> within each <url>, with pages from one host. This tutorial adds three publishing checks:
- Each input sitemap must be nonempty, with exactly one plain-text
<loc>per entry. - Every URL must use the scheme and host (including an explicit port) specified by
--origin. Whitespace, backslashes, and fragments are rejected; query strings are allowed. - Check exact URL strings for duplicates across all input files, and require every
--requireURL to appear at least once.
Duplicate comparison does not merge different percent encodings, query parameter orders, or trailing slashes. The script does not recurse through indexes, make network requests, validate the full XML Schema, check lastmod or hreflang relationships, confirm search indexing, or check local HTML files. It parses one XML file at a time and retains a set of seen URLs, making it suitable for small or medium sites you build yourself. Do not use it for untrusted uploads.
2. Setup: create a repeatable fixture
Use an existing Python 3.9 or newer installation; no third-party packages are needed. Commands assume bash/zsh on Linux. These examples were verified with Python 3.11.4 and Hugo 0.164.0.
- Create a directory:
python3 --version
mkdir sitemap-check-demo
cd sitemap-check-demo
- Save this as
demo.xml. Thexhtml:linkis a translation link. The script counts only the direct<loc>under each<url>, so translation links do not inflate the total.
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:xhtml="http://www.w3.org/1999/xhtml">
<url>
<loc>https://example.org/guide.html</loc>
<xhtml:link rel="alternate" hreflang="en"
href="https://example.org/en/guide.html" />
</url>
<url>
<loc>https://example.org/en/guide.html</loc>
</url>
</urlset>
- Save the full implementation below as
check_sitemap.pyin the same directory.
3. Implementation: read loc with its namespace
"""Check local urlset files, exact duplicate URLs, and required pages. Python 3.9+."""
import argparse
import json
from pathlib import Path
import sys
from urllib.parse import urlsplit
import xml.etree.ElementTree as ET
NS = "{http://www.sitemaps.org/schemas/sitemap/0.9}"
def origin_arg(value):
try:
parts = urlsplit(value)
parts.port # Validate a supplied port.
if (parts.scheme not in ("http", "https") or not parts.hostname
or parts.username is not None or parts.password is not None
or parts.path not in ("", "/") or parts.query or parts.fragment
or any(ord(c) < 33 or c.isspace() for c in value)
or "\\" in value or "?" in value or "#" in value):
raise ValueError("expected an HTTP(S) origin with no path or credentials")
except ValueError as exc:
raise argparse.ArgumentTypeError(str(exc)) from exc
return parts.scheme, parts.netloc
def check_url(value, origin):
if (not value or any(ord(c) < 33 or c.isspace() for c in value)
or "\\" in value or "#" in value):
raise ValueError(f"empty URL, whitespace, backslash, or fragment: {value!r}")
parts = urlsplit(value)
if (parts.scheme, parts.netloc) != origin or not parts.path.startswith("/"):
raise ValueError(f"URL must use the configured origin and an absolute path: {value!r}")
return value
def collect(paths, origin):
seen = set()
for path in paths:
root = ET.parse(path).getroot()
if root.tag == NS + "sitemapindex":
raise ValueError(f"{path}: sitemap index; pass its child urlset files")
if root.tag != NS + "urlset":
raise ValueError(f"{path}: expected urlset with the sitemap namespace")
if len(root) == 0:
raise ValueError(f"{path}: empty urlset")
for number, entry in enumerate(root, 1):
locations = entry.findall(NS + "loc")
if (entry.tag != NS + "url" or len(locations) != 1
or len(locations[0]) != 0):
raise ValueError(f"{path}: entry {number}: expected url with one text loc")
value = check_url(locations[0].text or "", origin)
if value in seen:
raise ValueError(f"{path}: duplicate URL: {value}")
seen.add(value)
return seen
def main():
parser = argparse.ArgumentParser(description=__doc__, allow_abbrev=False)
parser.add_argument("sitemaps", nargs="+", type=Path)
parser.add_argument("--origin", required=True, type=origin_arg)
parser.add_argument("--require", action="append", default=[], metavar="URL")
args = parser.parse_args()
try:
required = {check_url(value, args.origin) for value in args.require}
urls = collect(args.sitemaps, args.origin)
missing = required - urls
if missing:
raise ValueError("missing required URLs: " + ", ".join(sorted(missing)))
except (OSError, ET.ParseError, ValueError) as exc:
print(f"ERROR: {exc}", file=sys.stderr)
return 1
print(json.dumps({"files": len(args.sitemaps), "required": len(required),
"urls": len(urls)}, sort_keys=True))
return 0
if __name__ == "__main__":
sys.exit(main())
The ElementTree namespace documentation explains that parsed tags use the form {namespace}tag. Consequently, findall("url") may find nothing. This script matches NS + "url" and NS + "loc" explicitly, and reports a missing namespace as a failure.
The seen set spans input files, detecting duplicates both within a language sitemap and between sitemaps. --require records your explicit release requirements: a sitemap alone cannot tell the script which unpublished articles should have appeared.
4. Usage and validation
- Inspect help and check both required URLs:
python3 check_sitemap.py --help
python3 check_sitemap.py demo.xml --origin https://example.org \
--require https://example.org/guide.html \
--require https://example.org/en/guide.html
Expect this JSON and exit status 0:
{"files": 1, "required": 2, "urls": 2}
- Test duplicate input, a missing page, and the wrong origin. Each command should write an error to stderr, exit with
1, and leave stdout without success JSON:
python3 check_sitemap.py demo.xml demo.xml --origin https://example.org
printf 'exit=%s\n' "$?"
python3 check_sitemap.py demo.xml --origin https://example.org \
--require https://example.org/missing.html
printf 'exit=%s\n' "$?"
python3 check_sitemap.py demo.xml --origin https://wrong.example.org
printf 'exit=%s\n' "$?"
The errors respectively contain duplicate URL, missing required URLs, and configured origin. Invalid CLI arguments, such as omitting --origin, return 2 through argparse.
- Save this as
check_cases.pyand runpython3 check_cases.py. It also checks a missing namespace, a missing loc, accidental index input, truncated XML, and unchanged inputs:
from pathlib import Path
import subprocess
import sys
import tempfile
script = Path("check_sitemap.py").resolve()
original = Path("demo.xml").read_text(encoding="utf-8")
ns = ' xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"'
cases = [
(original, 0, '"urls": 2'),
(original.replace(ns, ""), 1, "sitemap namespace"),
(original.replace("<loc>", "<missing>").replace("</loc>", "</missing>"),
1, "one text loc"),
(original.replace("/en/guide.html</loc>", "/guide.html</loc>"),
1, "duplicate URL"),
('<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"/>',
1, "child urlset files"),
('<urlset', 1, "ERROR:"),
]
with tempfile.TemporaryDirectory(prefix="sitemap-tests-") as directory:
path = Path(directory) / "case.xml"
for text, expected, marker in cases:
path.write_text(text, encoding="utf-8")
result = subprocess.run(
[sys.executable, str(script), str(path),
"--origin", "https://example.org"],
capture_output=True, text=True, timeout=5,
)
assert result.returncode == expected, result.stderr
assert marker in (result.stdout if expected == 0 else result.stderr)
if expected:
assert result.stdout == ""
assert path.read_text(encoding="utf-8") == text
print("OK: namespace, loc, duplicates, index, and malformed XML")
Expected output:
OK: namespace, loc, duplicates, index, and malformed XML
5. Add a temporary Hugo preview
Save the checker in your Hugo project root. This monolingual example assumes a baseURL of https://example.org/ and a page that should publish at /guide.html. First replace the origin and required page with your project’s real values.
preview_root="$(mktemp -d)"
(
trap 'rm -rf -- "$preview_root"' EXIT
hugo --destination "$preview_root/site" &&
python3 check_sitemap.py "$preview_root/site/sitemap.xml" \
--origin https://example.org \
--require https://example.org/guide.html
)
--destination selects Hugo’s build output. The command creates a separate directory and checks only after a successful build. On exit, the subshell deletes only the preview directory just created. If your project uses a named configuration file, add its actual --config argument after hugo. This example checks output without deploying it.
For a multilingual project, first open sitemap.xml in the preview root and inspect each <sitemap><loc> child file location. Inside the subshell above, replace the monolingual checker command with the following. These paths are an example for a Chinese/English project; adjust them to the actual index, and use this approach only when the languages share an origin:
python3 check_sitemap.py \
"$preview_root/site/zh-cn/sitemap.xml" \
"$preview_root/site/en/sitemap.xml" \
--origin https://example.org \
--require https://example.org/guide.html \
--require https://example.org/en/guide.html
Do not read .//loc from an index and treat those results as article URLs. A language sitemap stored at /zh-cn/sitemap.xml does not imply that its article URLs must start with /zh-cn/.
6. Troubleshooting
| Error | Action |
|---|---|
sitemap index; pass its child urlset files |
Open the index and pass the corresponding local child sitemap paths. The script does not download their URLs. |
expected urlset with the sitemap namespace |
Confirm the input is an XML page sitemap. Keep the protocol namespace in custom templates; removing the check would hide empty results. |
expected url with one text loc |
Fix the custom template so each entry has exactly one direct, plain-text loc. |
duplicate URL |
Search source front matter and permalink rules for the reported URL. Resolve the collision and rebuild into a new directory; editing output XML does not fix source configuration. |
missing required URLs |
Confirm the required URL matches the actual permalink, then use the missing Hugo post guide to inspect drafts and publication dates. |
configured origin |
Compare the scheme, hostname, explicit port, and www in baseURL and --origin. Keep the host check to expose incorrect addresses. |
After the preview passes, use your existing publishing workflow and curl deployment checks to verify actual page content. A URL in a sitemap does not establish HTTP availability or guarantee search indexing.
7. Summary
Read page sitemaps with namespace-aware XML queries and check duplicates across files to expose template and permalink errors before release. List the pages required for this release with --require, and verify failure exit statuses first. Publish after the checker passes against an independent preview directory.
- Author: yanzhi
- Original article: https://www.bobobk.com/en/hugo-sitemap-check-python.html
- License: This work is licensed under CC BY-NC-ND 4.0. See the license for attribution, noncommercial use and no-derivatives terms.
Related articles
- Missing Hugo Posts: Diagnose Drafts, Publication Dates, Expiry, and Output Paths
- Check Hugo Internal Links and Images with Python Before Publishing
- Troubleshoot Linux Scheduled Jobs: Python Paths, Working Directories, and Environment
- Read the Last N Log Lines in Python with deque
- Find Duplicate Files with Python: Size Filters, Chunked SHA-256, and Read-Only Reports