Troubleshoot Linux Scheduled Jobs: Python Paths, Working Directories, and Environment

A Python script works in your terminal but fails as a scheduled job with FileNotFoundError, python3: not found, or missing configuration. It may depend on your current directory, your terminal’s PATH, or a virtual environment you activated earlier.

This beginner-to-intermediate tutorial reproduces each problem with a three-record task, then builds a predictable entry script. You do not need a running cron daemon or a crontab edit: first run the command in a minimal environment, then connect it to your existing scheduler.

1. Method: put the required settings in the entry script

The official Cronie crontab manual states that jobs run as the crontab owner, with /bin/sh as the default SHELL and account-based HOME and LOGNAME settings. Do not assume this loads your interactive shell startup configuration. Other schedulers may provide different environments.

We explicitly choose the project working directory, Python interpreter path, and required variables. Print only the few settings needed for diagnosis rather than dumping an environment that may contain credentials.

2. Setup

Use Linux, Python 3.9 or newer with its venv module, GNU env, and /bin/sh. The examples were verified on Linux with Python 3.11.4. Hugo is only needed for section 7; there are no third-party Python dependencies.

1. Create a fresh demo directory in bash or zsh. Use a new directory name when repeating the experiment. Run subsequent commands in the same terminal to keep the job_root variable:

python3 --version
command -v python3
command -v env
env --help
mkdir cron-env-demo
cd cron-env-demo
job_root="$PWD"
python3 -m venv --without-pip .venv
printf 'alpha\nbeta\ngamma\n' > data.txt
mkdir empty-bin

--without-pip creates a local virtual environment without installing packages. If venv is unavailable, use an existing project environment or ask your administrator to provide the module. Do not change the system Python default symlink.

2. Save the next section’s code as job.py, or download job.py.

3. Write a Python task with useful diagnostics

"""Inspect the small set of settings this demo needs. Python 3.9+."""
import os
from pathlib import Path
import sys


def main():
    print(f"cwd={Path.cwd()}", flush=True)
    print(f"python={sys.executable}", flush=True)
    print(f"venv={sys.prefix != sys.base_prefix}", flush=True)
    print(f"PATH={os.environ.get('PATH', '<unset>')}", flush=True)
    try:
        rows = Path("data.txt").read_text(encoding="utf-8").splitlines()
    except OSError as exc:
        print(f"ERROR: {exc}", file=sys.stderr)
        return 1
    if os.environ.get("REPORT_MODE") != "daily":
        print("ERROR: REPORT_MODE must be daily", file=sys.stderr)
        return 2
    count = sum(bool(row.strip()) for row in rows)
    print(f"OK: records={count}; mode=daily", flush=True)
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Run it from the project directory first:

REPORT_MODE=daily "$job_root/.venv/bin/python" "$job_root/job.py"

venv=True confirms that the actual interpreter belongs to a virtual environment. The final line should be OK: records=3; mode=daily.

Python’s venv documentation explains that you can invoke the environment’s interpreter by its full path without activation. Check sys.prefix != sys.base_prefix; do not rely solely on the VIRTUAL_ENV variable.

4. Reproduce three separate errors

Error 1: the script exists, but its input cannot be found

(
  cd /
  env -i PATH=/usr/bin:/bin REPORT_MODE=daily \
    "$job_root/.venv/bin/python" "$job_root/job.py"
)
printf 'exit=%s\n' "$?"

The output shows cwd=/, a missing data.txt, and finally exit=1. An absolute script path does not change the working directory. Relative input paths are still resolved from the current directory.

Error 2: PATH does not contain the interpreter

env -i PATH="$job_root/empty-bin" python3 --version
printf 'exit=%s\n' "$?"

This deliberately searches only an empty directory. Expect a missing python3 error and exit=127. It isolates executable lookup; it does not imply that every cron installation fails to find Python.

Error 3: a setting only existed in the interactive terminal

You are still in the project directory and the input exists, but this invocation omits REPORT_MODE:

env -i PATH=/usr/bin:/bin \
  "$job_root/.venv/bin/python" "$job_root/job.py"
printf 'exit=%s\n' "$?"

Expect ERROR: REPORT_MODE must be daily and exit=2.

The official GNU env documentation defines -i as starting with an empty environment before applying command-line assignments. This experiment is stricter than many real scheduler environments. It reveals hidden dependencies without simulating the scheduler’s account, permissions, timezone, or service restrictions.

5. Fix: choose the directory, interpreter, and variables

Save this as run_job.sh beside job.py, or download run_job.sh. Invoke its actual path rather than a symlink located elsewhere.

#!/bin/sh
set -eu
PATH=/usr/bin:/bin
export PATH
job_dir=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
cd "$job_dir"
REPORT_MODE=daily
export REPORT_MODE
exec "$job_dir/.venv/bin/python" -u "$job_dir/job.py"

The entry script first sets a trusted utility search path, finds its own project directory, changes directory, supplies the task’s nonsensitive setting, and runs the local interpreter. exec makes Python’s exit status the entry script’s status; -u sends output promptly to the redirected log.

REPORT_MODE is not a secret. For real credentials, keep your deployment’s existing secure injection method rather than copying secrets into a script or article.

6. Validate the fixed entry script

Start from the root directory with an empty environment and capture both output streams in a demo log:

(
  cd /
  env -i PATH=/usr/bin:/bin /bin/sh "$job_root/run_job.sh"
) > "$job_root/job.log" 2>&1
job_status=$?
cat "$job_root/job.log"
printf 'exit=%s\n' "$job_status"

cwd should point to your demo directory, and python to its .venv/bin/python. The final output should be:

OK: records=3; mode=daily
exit=0

Repeating the command should still count three records. The demo overwrites its log with >; for retained application logs, see the Python log rotation tutorial. Once the entry script is reliable, use it as the command in your existing scheduler and verify permissions as the actual job account.

If Python launches another program, use the subprocess timeout and logging tutorial to check its exit status and business output separately.

7. Apply the same method to a temporary Hugo build

Hugo may also live in a directory only your terminal adds to PATH. The commands below require hugo to be available in your current terminal. Record its full path, create a minimal local site, then build from / into a fresh temporary directory:

command -v hugo
hugo version
hugo --help
hugo_bin="$(command -v hugo)"
mkdir -p mini-site/content mini-site/layouts
printf 'baseURL = "https://example.org/"\n' > mini-site/hugo.toml
printf '# Scheduled preview\n' > mini-site/content/_index.md
cat > mini-site/layouts/index.html <<'HTML'
<!doctype html><html lang="en"><head><title>Demo</title></head><body>{{ .Content }}</body></html>
HTML
preview_root="$(mktemp -d)"
(
  cd /
  env -i PATH=/usr/bin:/bin "$hugo_bin" \
    --source "$job_root/mini-site" \
    --cacheDir "$preview_root/cache" \
    --destination "$preview_root/site"
)
test -s "$preview_root/site/index.html"
cat "$preview_root/site/index.html"

The final HTML should contain Scheduled preview. This example was verified with Hugo 0.164.0. It does not publish a site; mktemp creates a new output directory each time.

The official Hugo command reference lists --source, --cacheDir, and --destination. Use them to preview your real project before connecting a publishing workflow. For builds in GitHub Actions, see the Hugo build and deployment tutorial.

8. Troubleshooting

Validation covered the three failures, successful and repeated entry-script runs, and preservation of a nonzero status when the input file was missing. After the minimal-environment test passes, check actual scheduled-run logs too. No real cron configuration was tested or changed for this tutorial.

9. Summary

Record the working directory, interpreter, and required variables, then reproduce failures in a minimal environment. Put the verified settings in a fixed entry script to reduce dependence on interactive terminal state. Preserve exit statuses and keep checking result files in both publishing and batch workflows.