Fix Linux Script Startup Errors: Diagnose CRLF and BOM with Python

A script copied from Windows exists on Linux, but running it reports bad interpreter, No such file or directory, or /usr/bin/env: 'python3\r': No such file or directory. Invisible bytes on the first line may be responsible. Repeating chmod +x does not change those bytes.

This beginner-to-intermediate tutorial uses the Python standard library to reproduce startup failures caused by CRLF line endings and a UTF-8 BOM, inspect the bytes, create a new LF copy, and verify execution. Use the method for shell scripts, Python automation entry points, and Hugo build wrappers you maintain. The experiment only touches a new demo directory.

1. Method: distinguish startup failures from program failures

For direct script execution, Linux expects the file to start with #!. The Linux kernel’s script loader separates the first line at LF and trims trailing spaces and tabs. This implies that \r in #!/bin/sh\r\n becomes part of the interpreter path, while a BOM before #! prevents script-format recognition.

These symptoms are not exclusive to one cause. An interactive shell may fall back to interpreting the file, changing the error you see. We start files directly from Python below to avoid shell fallback hiding the original execution error. If a script works interactively but fails on a schedule, also check scheduled Python job paths and environments.

2. Setup and fixtures

Use Linux, Python 3.9 or newer, /bin/sh, and chmod. There are no third-party dependencies, and dos2unix is unnecessary. The examples were tested on Linux with Python 3.11.4.

1. Create a fresh directory in bash or zsh, choosing a new name when repeating the experiment. Stay in this directory for subsequent steps:

python3 --version
mkdir script-bytes-demo
cd script-bytes-demo
python3 - <<'PY_DATA'
import codecs
from pathlib import Path

script = b"#!/bin/sh\nprintf 'job ok\\n'\n"
Path("good.sh").write_bytes(script)
Path("bad-crlf.sh").write_bytes(script.replace(b"\n", b"\r\n"))
Path("bad-bom.sh").write_bytes(codecs.BOM_UTF8 + script)
Path("bare-cr.sh").write_bytes(b"#!/bin/sh\rprintf 'stop'\r")
for name in ("good.sh", "bad-crlf.sh", "bad-bom.sh"):
    Path(name).chmod(0o700)
PY_DATA

2. good.sh uses LF, bad-crlf.sh uses CRLF, and bad-bom.sh starts with a UTF-8 BOM. bare-cr.sh contains standalone CR bytes to test refusal of an ambiguous conversion. The generated executable scripts only print one line.

3. Inspect raw bytes to locate the problem

Run this diagnostic. repr() exposes hidden \r bytes and the BOM:

python3 - <<'PY_DIAG'
from pathlib import Path
import subprocess

for name in ("good.sh", "bad-crlf.sh", "bad-bom.sh"):
    raw = Path(name).read_bytes()
    print(name, "first_line=", repr(raw.split(b"\n", 1)[0]), flush=True)
    try:
        result = subprocess.run(["./" + name], capture_output=True, check=False)
    except OSError as exc:
        print("exec failed:", exc.errno, exc.strerror)
    else:
        print("exit=", result.returncode, "stdout=", repr(result.stdout))
PY_DIAG

Output on the test machine follows. Error wording can change with the system language; the error numbers remain useful:

good.sh first_line= b'#!/bin/sh'
exit= 0 stdout= b'job ok\n'
bad-crlf.sh first_line= b'#!/bin/sh\r'
exec failed: 2 No such file or directory
bad-bom.sh first_line= b'\xef\xbb\xbf#!/bin/sh'
exec failed: 8 Exec format error

Here, 2 and 8 are OSError.errno values, not child-process exit codes: startup failed before there was a subprocess result. Avoid read_text().splitlines() for this diagnostic; text reading can translate line endings and conceal the original CRLF. Python’s binary reading interface preserves file bytes. The open() documentation describes newline translation in text mode.

4. Complete implementation: check or create an LF copy

Save this code as normalize_script.py in the current directory. Save the tool itself with UTF-8 encoding and LF line endings:

"""Check UTF-8 script line endings, or write a new LF copy. Python 3.9+."""
import argparse
import codecs
from pathlib import Path
import sys

MAX_BYTES = 1024 * 1024


def inspect_script(path):
    if not path.is_file():
        raise ValueError("input must be a regular file")
    with path.open("rb") as source:
        raw = source.read(MAX_BYTES + 1)
    if len(raw) > MAX_BYTES:
        raise ValueError("input exceeds 1 MiB")
    bom = raw.startswith(codecs.BOM_UTF8)
    body = raw[len(codecs.BOM_UTF8):] if bom else raw
    body.decode("utf-8", errors="strict")
    if b"\x00" in body:
        raise ValueError("NUL byte: expected a UTF-8 text script")
    if not body.startswith(b"#!"):
        raise ValueError("expected #! at the start after an optional UTF-8 BOM")
    crlf = body.count(b"\r\n")
    bare_cr = body.count(b"\r") - crlf
    return body, bom, crlf, bare_cr


def main():
    parser = argparse.ArgumentParser(description=__doc__, allow_abbrev=False)
    parser.add_argument("source", type=Path)
    parser.add_argument("--output", type=Path,
                        help="create a new LF copy; never overwrite a path")
    args = parser.parse_args()
    try:
        body, bom, crlf, bare_cr = inspect_script(args.source)
        print(f"bom={int(bom)} crlf={crlf} bare_cr={bare_cr}")
        if args.output is None:
            return 1 if bom or crlf or bare_cr else 0
        if bare_cr:
            raise ValueError("bare CR found; inspect the source manually")
        cleaned = body.replace(b"\r\n", b"\n")
        with args.output.open("xb") as target:
            target.write(cleaned)
        print(f"WROTE: {args.output}")
        return 0
    except (OSError, ValueError, UnicodeError) as exc:
        print(f"ERROR: {exc}", file=sys.stderr)
        return 1


if __name__ == "__main__":
    raise SystemExit(main())

The tool reads at most 1 MiB and requires UTF-8 text starting with #! after an optional BOM. It handles only these small scripts, does not detect other encodings, and does not check whether an interpreter exists. Input symlinks are followed to their targets. Output uses exclusive binary creation mode xb, so existing files and links cannot be overwritten.

Check mode writes no files and returns 1 for a BOM, CRLF, or standalone CR. With --output, the tool removes an initial UTF-8 BOM and converts CRLF to LF. It refuses standalone CR rather than guessing its meaning. Python’s codec documentation describes the UTF-8 BOM; this example handles its bytes directly without converting arbitrary encodings.

5. Usage: inspect first, then validate the copy

1. Show help and repair the CRLF demo file:

python3 normalize_script.py --help
if python3 normalize_script.py bad-crlf.sh; then
  printf 'Already clean\n'
else
  printf 'Check reported a problem; inspect before repairing\n'
fi
python3 normalize_script.py bad-crlf.sh --output repaired.sh
python3 normalize_script.py repaired.sh
/bin/sh -n repaired.sh
chmod u+x repaired.sh
./repaired.sh

Checking the original reports bom=0 crlf=2 bare_cr=0 and returns 1. Successfully creating the copy returns 0. Checking the copy should report bom=0 crlf=0 bare_cr=0, and executing it should print job ok. Invalid arguments return 2; input, decoding, and write failures return 1. A diagnostic may already be on standard output, so also inspect the exit code and standard error.

2. Apply the same procedure to the BOM file:

python3 normalize_script.py bad-bom.sh --output repaired-bom.sh
python3 normalize_script.py repaired-bom.sh
/bin/sh -n repaired-bom.sh
/bin/sh repaired-bom.sh

New files do not inherit the source’s execute permission. Apply chmod u+x explicitly after reviewing content and syntax. /bin/sh -n checks syntax only; it cannot establish correct runtime behavior. Execution above is limited to our known demo scripts. Review the diff and follow project validation procedures after repairing production scripts instead of running tasks with side effects merely to diagnose them.

Converting CRLF to LF also changes literal CR bytes inside multiline strings or here-documents. Use this tool only after confirming those bytes represent line endings; handle intentionally embedded data manually. An interrupted or failed write can leave an incomplete new copy. Keep the original and retry with a fresh output name.

6. Validate failure branches and source preservation

Run this self-check in the same demo directory. It additionally creates fixed-crlf.sh and fixed-bom.sh; use an entirely fresh demo directory before repeating it:

python3 - <<'PY_CHECK'
import codecs
from pathlib import Path
import subprocess
import sys


def run(*arguments):
    return subprocess.run(
        [sys.executable, "normalize_script.py", *arguments],
        capture_output=True, text=True, check=False,
    )


assert run("--help").returncode == 0
assert run("good.sh").returncode == 0
for name in ("bad-crlf.sh", "bad-bom.sh", "bare-cr.sh"):
    result = run(name)
    assert result.returncode == 1, result

for source, output in (("bad-crlf.sh", "fixed-crlf.sh"),
                       ("bad-bom.sh", "fixed-bom.sh")):
    before = Path(source).read_bytes()
    result = run(source, "--output", output)
    assert result.returncode == 0, result
    assert Path(source).read_bytes() == before
    cleaned = Path(output).read_bytes()
    assert cleaned == Path("good.sh").read_bytes()
    assert not cleaned.startswith(codecs.BOM_UTF8) and b"\r" not in cleaned
    result = run(source, "--output", output)
    assert result.returncode == 1, result
    assert Path(output).read_bytes() == cleaned
    result = subprocess.run(["/bin/sh", output], capture_output=True, check=False)
    assert result.returncode == 0 and result.stdout == b"job ok\n", result

assert run("bare-cr.sh", "--output", "refused.sh").returncode == 1
assert not Path("refused.sh").exists()
assert run("missing.sh").returncode == 1
assert run(".").returncode == 1
assert run("good.sh", "--unknown").returncode == 2
print("OK: detection, repair, unchanged sources, no overwrite, errors")
PY_CHECK

Expected final output:

OK: detection, repair, unchanged sources, no overwrite, errors

The tests check detection, exact repaired bytes, unchanged sources, refusal to overwrite output, errors for missing and directory inputs, and argument-error exit code 2. Do not attach || true to every check, because that hides real failures. For calling business scripts from Python afterward, see external-command timeouts and error handling.

7. Troubleshoot and prevent recurrence

1. Save scripts in your editor as UTF-8 without BOM, with LF line endings.
2. For Git projects, review and merge these rules into the existing project .gitattributes instead of overwriting its contents:

*.sh text eol=lf
*.py text eol=lf

Git’s official documentation explains text normalization and eol=lf checkout behavior. These rules do not remove BOMs or immediately rewrite existing working-tree files. Repair and review affected scripts first, then inspect the attributes applying to an actual path:

git check-attr text eol -- scripts/job.sh

Replace the path with a script in your repository. This example requires no staging, commits, or global Git configuration changes. After repairing a Hugo build wrapper, also validate links and images in the output; see checking Hugo internal links with Python.

8. Summary

When an existing script cannot start, inspect the shebang’s raw bytes before distinguishing CRLF, BOM, missing interpreters, and permission problems. Convert into a new copy, verify its exact contents, and retain the source for comparison. Set your editor to UTF-8 without BOM and LF to reduce repeat startup failures.