Fix Python UnicodeEncodeError in Linux Jobs: Inspect stdout and Select UTF-8

A script prints Chinese correctly in a terminal, then fails with UnicodeEncodeError: 'ascii' codec can't encode characters when a scheduled job redirects its logs. Inspect the failing process’s actual standard-output encoding, then configure UTF-8 at launch and verify the bytes written.

This tutorial suits beginner-to-intermediate readers troubleshooting standard streams on Linux. The interpreter and working directory can differ too; check the entry point with the scheduled-job path guide.

1. Setup: distinguish three encodings

Use Linux, Bash, and Python 3.11 or newer. No third-party packages are needed. The examples were tested on Linux with Python 3.11.4. Choose a fresh directory name when repeating the exercise:

python3 --version
mkdir unicode-stdio-demo
cd unicode-stdio-demo

Do not infer the result from LANG alone: UTF-8 mode and standard-stream overrides also matter. We check terminal output, ordinary text files, and input data separately; changing one does not automatically fix the others.

The official Python 3.15 release notes document UTF-8 mode being enabled by default in 3.15. Older interpreters and explicit overrides can still use other encodings, so our reproduction deliberately disables automatic behavior. It does not imply that every scheduled job fails.

2. Diagnose: inspect the actual standard streams

1. Save this as encoding_probe.py:

import json
import locale
import os
import sys

print(json.dumps({
    "python": sys.executable,
    "utf8_mode": sys.flags.utf8_mode,
    "locale_encoding": locale.getencoding(),
    "stdout_encoding": sys.stdout.encoding,
    "stdout_errors": sys.stdout.errors,
    "stderr_encoding": sys.stderr.encoding,
    "LC_ALL": os.environ.get("LC_ALL"),
    "LANG": os.environ.get("LANG"),
    "PYTHONIOENCODING": os.environ.get("PYTHONIOENCODING"),
}, ensure_ascii=True))

2. Run it in the terminal and with redirection. For a production failure, also run the same probe through the failing task’s existing entry point:

python3 encoding_probe.py
python3 encoding_probe.py > encoding.json
cat encoding.json

JSON uses ensure_ascii=True to escape non-ASCII paths so the diagnostic itself can survive ASCII output. Compare python, utf8_mode, and stdout_encoding. locale_encoding describes the system locale and can still report ASCII with UTF-8 mode enabled. See the Python sys documentation for standard-stream attributes.

3. Save the minimal reproduction as hello.py. Unicode escapes keep the source file’s save encoding out of the experiment:

print("\u5b8c\u6210: 2 rows")

4. Apply ASCII settings to these commands only, leaving system locale and scheduler settings untouched:

env LC_ALL=C PYTHONCOERCECLOCALE=0 PYTHONUTF8=0 \
  PYTHONIOENCODING=ascii:strict python3 encoding_probe.py
if env LC_ALL=C PYTHONCOERCECLOCALE=0 PYTHONUTF8=0 \
  PYTHONIOENCODING=ascii:strict python3 hello.py > bad.log; then
  printf 'Unexpected success\n'
else
  printf 'exit=%s\n' "$?"
fi

The probe should show stdout_encoding: "ascii" and utf8_mode: 0. The traceback then contains UnicodeEncodeError, the final line is exit=1, and bad.log is empty. The if handles the expected failure, including in entry scripts using set -e.

3. Method 1: configure UTF-8 standard streams

When the log receiver expects UTF-8, add an override to the Python launch command:

env LC_ALL=C PYTHONCOERCECLOCALE=0 PYTHONUTF8=0 \
  PYTHONIOENCODING=utf-8:strict python3 hello.py > stdio.log

A successful run exits with code 0. stdio.log contains 完成: 2 rows followed by a newline. Open the log as UTF-8; a viewer using a different decoding can still display garbled text.

PYTHONIOENCODING configures standard input, output, and error encodings before interpreter startup. Here strict rejects invalid conversions on stdin/stdout; stderr retains backslashreplace error handling. Ordinary open() file encodings are separate. The -E and -I options ignore Python environment variables, so inspect launch arguments too.

Avoid errors="ignore", which can silently remove business text. Setting os.environ["PYTHONIOENCODING"] after startup also does not rebuild existing standard streams.

4. Method 2: enable UTF-8 mode and specify file encodings

1. To select UTF-8 defaults for the Python entry point, use -X utf8. First remove a potentially conflicting standard-stream override:

env -u PYTHONIOENCODING LC_ALL=C PYTHONCOERCECLOCALE=0 \
  python3 -X utf8 hello.py > mode.log

Python UTF-8 mode also affects default text-file and filesystem encodings. Configure it at startup. PYTHONIOENCODING=ascii can still override standard streams, so -X utf8 alone cannot defeat that conflict. The mode does not configure external commands or log viewers.

2. For application files, specify the encoding according to the data contract. Save this as report.py:

from pathlib import Path

source = Path("input.txt")
with source.open("r", encoding="utf-8", errors="strict") as handle:
    rows = sum(1 for line in handle if line.strip())
message = f"\u5b8c\u6210: {rows} rows\n"
Path("report.txt").write_text(message, encoding="utf-8", errors="strict")
print(message, end="")

3. Create two UTF-8 input lines and run the report:

python3 - <<'PY_INPUT'
from pathlib import Path
Path("input.txt").write_bytes("\u7532\n\u4e59\n".encode("utf-8"))
PY_INPUT
env LC_ALL=C PYTHONCOERCECLOCALE=0 PYTHONUTF8=0 \
  PYTHONIOENCODING=utf-8:strict python3 report.py > report.log

This deliberately leaves UTF-8 mode disabled and fixes only standard streams. File operations still succeed because the code explicitly specifies encoding="utf-8". See the open() documentation for default encoding behavior. If the source is GB18030, UTF-16, or a CSV with a BOM, use the CSV encoding guide to establish its format rather than assuming all bytes are UTF-8.

5. Validate: check exit codes and actual bytes

After creating the files and running the preceding commands, execute this self-check from the demo directory. It launches the current interpreter and captures bytes so the parent’s text-decoding settings cannot interfere with assertions:

python3 - <<'PY_CHECK'
import json
import os
from pathlib import Path
import subprocess
import sys

base = os.environ.copy()
base.update(LC_ALL="C", PYTHONCOERCECLOCALE="0", PYTHONUTF8="0",
            PYTHONIOENCODING="ascii:strict")
expected = "\u5b8c\u6210: 2 rows\n".encode("utf-8")

def run(args, env):
    return subprocess.run([sys.executable, *args], env=env,
                          capture_output=True, check=False, timeout=10)

probe = run(["encoding_probe.py"], base)
assert probe.returncode == 0, probe.stderr
info = json.loads(probe.stdout)
assert info["stdout_encoding"] == "ascii" and info["utf8_mode"] == 0, info
bad = run(["hello.py"], base)
assert bad.returncode != 0 and b"UnicodeEncodeError" in bad.stderr, bad
assert bad.stdout == b"", bad.stdout
stdio = dict(base, PYTHONIOENCODING="utf-8:strict")
fixed = run(["hello.py"], stdio)
assert fixed.returncode == 0 and fixed.stdout == expected, fixed
mode_env = base.copy()
mode_env.pop("PYTHONIOENCODING")
mode = run(["-X", "utf8", "hello.py"], mode_env)
assert mode.returncode == 0 and mode.stdout == expected, mode
override = run(["-X", "utf8", "hello.py"], base)
assert override.returncode != 0 and b"UnicodeEncodeError" in override.stderr
file_default = run(["-c", "from pathlib import Path; Path('input.txt').read_text()"], stdio)
assert file_default.returncode != 0 and b"UnicodeDecodeError" in file_default.stderr
report = run(["report.py"], stdio)
assert report.returncode == 0 and report.stdout == expected, report
assert Path("report.txt").read_bytes() == expected
for filename in ("stdio.log", "mode.log", "report.log"):
    assert Path(filename).read_bytes() == expected, filename
print("PASS: ASCII failure, both fixes, override precedence, file encoding, UTF-8 bytes")
PY_CHECK

Expect the final PASS: ... line. The checks cover ASCII output failure, both fixes, conflicting override precedence, and a file-decoding failure after standard streams are fixed. They also compare all three logs and the report byte for byte. They use only demo files and do not change production jobs.

6. Apply the fix and troubleshoot

Once the receiver is confirmed to accept UTF-8, apply the chosen launch method to the existing task entry point and keep its interpreter and working directory explicit. If logging.FileHandler writes the log, configure encoding="utf-8" separately; standard-stream overrides do not control it. See the official FileHandler documentation.

A source-file # coding: utf-8 declaration describes the source rather than print() output. This tutorial does not change the system locale. If other non-Python tools depend on locale settings, inspect their requirements separately.

7. Summary

Inspect standard-stream encodings through the failing entry point, then choose PYTHONIOENCODING or -X utf8 and remove conflicting overrides. Keep real file encodings explicit, and validate exit codes and output bytes before treating a visually normal log as proof of a fix.