Skip to content

Exit non-zero when the boot-time steamcmd update fails, before Oxide is installed - #1

Closed
compscidr wants to merge 14 commits into
zaroxh:masterfrom
compscidr:fix/fail-on-steamcmd-update-error
Closed

compscidr wants to merge 14 commits into
zaroxh:masterfrom
compscidr:fix/fail-on-steamcmd-update-error

Conversation

@compscidr

Copy link
Copy Markdown

Why

On 2026-09-03 the weekly and monthly servers on nas.local restarted for their scheduled wipe at 18:00 UTC — the moment Facepunch's monthly depot was flipping. steamcmd printed Error! App '258550' state is 0x486 after update job. and exited 0. start_rust.sh carried on, downloaded Oxide develop (2.0.7676, published 28 minutes earlier for the new build) over the old Rust install, and RustDedicated hung at boot:

MissingMethodException: Method not found: void Facepunch.ExceptionReporter.Initialize(string,string)

A hang is not an exit, so restart: unless-stopped never fired, the image's own update checker could not reach RCON to restart, and both servers stayed down for five days.

What

  • install_or_update accepts only steamcmd's own Success! App '258550' fully installed line (printed on every boot, including already-up-to-date). Anything else sleeps STEAMCMD_RETRY_DELAY (default 60s) and exits 1 before the Oxide step. The restart policy then retries until Steam serves the update, so a restart into a Facepunch rollout costs a few loops instead of an outage.
  • The sleep exists because docker resets its restart backoff for any container that ran over 10s; steamcmd takes ~20s to fail, so without it the loop would hit Steam every 20s.
  • STEAMCMD_SH and STEAMCMD_RETRY_DELAY let test/install_or_update_test.sh drive the extracted function with a stub steamcmd. Verified the test goes red when the guard is removed.
  • Workflow runs the test on PRs and gates the publish on it. README documents the behaviour and that a hand-rolled wrapper needs the same rule (rustd.xyz's onboarding text will say the same).

Rollout

Ships on the next latest build from main. The NAS role pulls the image, so the next nas.yml run recreates both containers — do it at a quiet hour.

🤖 Generated with Claude Code

https://claude.ai/code/session_01A39aALuGv7Pf3HCsbiqix8

compscidr and others added 14 commits October 22, 2024 19:42
… links to didstopia npm server which wasn't responding
RustDedicated now requires GLIBC >= 2.34 (2.32/2.33/2.35 for
UnityPlayer.so), but the ubuntu-20.04 base only ships glibc 2.31.
The server binary fails to load immediately after install, causing
containers to exit and restart-loop forever.

Mirrors upstream Didstopia/rust-server, which moved to the
nodejs-22-steamcmd-ubuntu-24.04 base and dropped the stale
package-lock files (regenerated by npm install at build time).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…digest

- Regenerate package-lock.json for all helper apps under node 22
  instead of deleting them, and switch builds to npm ci for
  reproducible dependency resolution
- Pin didstopia/base:nodejs-22-steamcmd-ubuntu-24.04 by manifest
  digest so rebuilds of the same commit get the same runtime
  (verified: image builds clean with these changes)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fix: update base image to nodejs-22-steamcmd-ubuntu-24.04 (fixes GLIBC restart loop)
The healthcheck runs rcon against localhost:$RUST_RCON_PORT, but that
env var only holds the image default (28016) — deployments that set
the port via the bind-mounted /etc/rust/rust.env (sourced by start.sh)
were always probed on the wrong port, so containers reported unhealthy
even with the server fully up. Source rust.env in the healthcheck
shell when present.

Verified against a live server: probe now exits 0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fix: source rust.env in healthcheck so rcon probes the real RCON port
…is installed

steamcmd exits 0 even when app_update fails. On 2026-09-03 the weekly and
monthly servers restarted for their scheduled wipe at 18:00 UTC, the moment
Facepunch's monthly depot was flipping; steamcmd printed "Error! App '258550'
state is 0x486 after update job." and exited 0, the script carried on,
installed an Oxide built for the new Rust (published 28 minutes earlier) over
the old build, and RustDedicated hung at boot with
MissingMethodException: Facepunch.ExceptionReporter.Initialize(string,string).
A hang is not an exit, so `restart: unless-stopped` never fired; both servers
stayed down for five days.

install_or_update now accepts only steamcmd's own success line (also printed
when already up to date). Anything else sleeps STEAMCMD_RETRY_DELAY (60s,
so the retry loop does not hit Steam every ~20s — docker resets its backoff
for a container that ran over 10s) and exits 1 before the Oxide step, so the
restart policy retries until Steam serves the update. STEAMCMD_SH lets the
new test stub the binary.

test/install_or_update_test.sh extracts the function and drives it with a
stub steamcmd printing the 0x486 line and the success line; the workflow
runs it on PRs and gates the publish on it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A39aALuGv7Pf3HCsbiqix8
Copilot AI lite review requested due to automatic review settings September 8, 2026 18:01
@compscidr

Copy link
Copy Markdown
Author

Opened against the wrong repository by mistake (meant for the compscidr fork). Sorry for the noise.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The workflow/publish path and startup script have correctness/operational issues (missing GH Actions permissions, non-atomic date tagging, unguarded env sourcing noise) that should be fixed before merging.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR hardens container startup so Rust is not started (and Oxide is not installed) when steamcmd fails to actually update Rust despite exiting 0, preventing a “new Oxide over old Rust” mismatch that can hang RustDedicated and defeat restart policies.

Changes:

  • Add a guard in install_or_update to require steamcmd’s Success! App '258550' fully installed line; otherwise sleep and exit non-zero so restart policies retry safely.
  • Add a shell test to pin the failure mode and gate publishing via GitHub Actions.
  • Update container build/runtime bits (base image/Node upgrade, npm ci, lockfile updates) and document new behavior and env-file overrides.
File summaries
File Description
start_rust.sh Enforces steamcmd success-line gating; adds env-file sourcing for overrides.
test/install_or_update_test.sh Adds regression test for “steamcmd exited 0 but update failed” behavior.
.github/workflows/docker-publish.yml Runs the new test on PRs and publishes from main.
README.md Documents env-file overrides and steamcmd failure handling behavior.
Dockerfile Updates base image/Node version and switches Node installs to npm ci; updates healthcheck env sourcing.
docker-compose.yml Adds compose example including env/seed file mounts.
rust.env Adds example rust env override file.
seed.env Adds example seed override file.
.gitignore Ignores IDE config and local rust-server data dir.
shutdown_app/package-lock.json Lockfile v3 / dependency resolution updates.
scheduler_app/package-lock.json Lockfile v3 format update.
restart_app/package-lock.json Lockfile v3 / dependency resolution updates.
rcon_app/package-lock.json Lockfile v3 / dependency resolution updates.
heartbeat_app/package-lock.json Lockfile update (incl. ws version bump) and metadata adjustments.
Review details

Files not reviewed (5)

  • heartbeat_app/package-lock.json: Generated file
  • rcon_app/package-lock.json: Generated file
  • restart_app/package-lock.json: Generated file
  • scheduler_app/package-lock.json: Generated file
  • shutdown_app/package-lock.json: Generated file
  • Files reviewed: 8/14 changed files
  • Comments generated: 8
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines 7 to 9
pull_request:
branches: [ "master" ]

env:
# Use docker.io for Docker Hub if empty
REGISTRY: ghcr.io
# github.repository as <account>/<repo>
IMAGE_NAME: ${{ github.repository }}


jobs:
Comment on lines +35 to +39
run: |
docker build . --tag ghcr.io/compscidr/rust-server:$(date +%m-%d-%Y)
docker tag ghcr.io/compscidr/rust-server:$(date +%m-%d-%Y) ghcr.io/compscidr/rust-server:latest
docker push ghcr.io/compscidr/rust-server:$(date +%m-%d-%Y)
docker push ghcr.io/compscidr/rust-server:latest No newline at end of file
Comment thread start_rust.sh
Comment on lines +9 to +13
set -o allexport
source /etc/rust/rust.env
# lets you split the seed into a separate config file to make wipes easier
source /etc/rust/seed.env
set +o allexport
Comment on lines +27 to +28
# this step requires a GH_TOKEN with read:packages write:packages and delete:packages
# tested with a classic token with the above permissions
Comment thread Dockerfile
Comment on lines +1 to +4
# nodejs-22-steamcmd-ubuntu-24.04 (Ubuntu 24.04, glibc 2.39, node 22) pinned by digest
FROM didstopia/base:nodejs-22-steamcmd-ubuntu-24.04@sha256:ae8f7bdab405f669eadcc7178627dd47def015e22227247547904b22156f5a47

LABEL maintainer="Didstopia <support@didstopia.com> & Zaroxh <admin@rusttilldawn.eu>"
LABEL org.opencontainers.image.source="https://github.com/compscidr/rust-server"
Comment thread README.md
the container will keep the same old values from when it was originally run.
This setup allows overriding anything passed in from docker. The config file
is located at `/etc/rust/rust.env` in the container. You can map a volume
locally to the file for peristence.
Comment thread start_rust.sh
exit_handler()
{
echo "Shutdown signal received"
echo "Shutdown signal received"
Comment on lines +15 to +24
run() {
local stub
stub=$(mktemp)
printf 'echo "%s"\nexit 0\n' "$1" > "$stub"
(
export STEAMCMD_SH=$stub STEAMCMD_RETRY_DELAY=0
eval "$(sed -n '/^install_or_update()/,/^}/p' start_rust.sh)"
install_or_update > /dev/null 2>&1
)
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants