LOCAL-VALIDATION.md

Local validation

Issue #135 introduces a complete local entry point and an opt-in signed configuration-evidence path. They run without GitHub Actions, gh, an Actions event, or an Actions API call. The established required-check policy remains in place until the parity and rollback gates below pass. The engine hosted adapter calls the complete local entry point and retains its independent main-policy guards. This implementation does not claim that host adoption or the standalone template migration has occurred.

Follow the operator walkthrough for machine setup, commit selection, complete runs, separate signer bootstrap, evidence provisioning, acceptance checks and recovery. It keeps host activation and policy cutover separate from repository validation.

Tools and execution boundary

Use a disposable Debian 13 or Ubuntu 24.04 Linux execution machine. Engine fixtures require root, runuser, mounted /proc and /dev/shm, Bash, Git with complete history, Python 3.11 or later, PyYAML, ShellCheck, OpenSSL with Ed25519 support, coreutils, util-linux, curl, jq, systemd 255 or newer and cgroup v2. The system systemd manager must be PID 1 on the execution machine. Both engine profiles check for jq before running stages because the installer fixtures require it. The full engine run also requires Docker Engine, Compose v2, the Docker Buildx plugin and access to its local Unix socket. The tool preflight checks Buildx before any validation stage so Docker cannot silently use its legacy builder for the APT-cache fixture image. The harness records tool versions. Missing tools, skipped tests and interrupted checks fail the run. Do not use an active fleet, application or signing host. The --disposable-host flag acknowledges this execution requirement; a filesystem check in both profiles also rejects standard controller, tester, deployer, status-receiver and signing-authority installation paths, installer metadata, and systemd unit markers. Signing paths include /etc/ci-fleet-signing, /var/lib/ci-fleet-signing and /srv/ci-fleet-signing. Any surviving directory, file or dangling symlink blocks candidate execution, including partial bootstrap or removal. That check cannot detect an unusually installed role, so the operator must verify the machine's role.

Configuration runs need the same prepared Linux/systemd worker, Bash, Git, Python 3.11 or later, ShellCheck and OpenSSL. The trusted launcher requires root; configuration stages execute as the locked, unprivileged ci-fleet-validation account without Docker, manager-bus or cgroup-control access. This is an explicit execution-host prerequisite, replacing the previous arbitrary unprivileged-Linux supervisor. There is no alternate process-tracking backend. macOS and Windows operators use a disposable Linux VM or a remote prepared execution worker.

After selecting the trusted engine checkout, prepare each fresh worker once:

sudo python3 -I /trusted/ci-fleet/scripts/local_validation_executor.py \
  --prepare --disposable-host

Preparation installs the committed executor and its revision/hash manifest under /usr/local/lib/ci-fleet-validation and creates a locked job account. It refuses an existing job account with a shared primary group or supplementary group memberships. Ordinary stages recheck that restriction before execution. It refuses uncommitted executor code. Consumers require the exact independently reviewed installed revision. There is no daemon, timer, fleet registration or signing credential. The entire worker remains disposable; rebuild it with the same reviewed prerequisites when lost. Do not run this setup on a fleet, application or signing host. The core configuration profile covers only the embedded configuration contract. Standalone-template and private-adopter suites additionally retain their own policy, updater, compatibility and release checks. Do not substitute the smaller core profile for those suites.

Populate complete repository history and tags before running. Policy is an independently reviewed engine commit, not a merge base chosen by the candidate. For trusted acceptance, run the orchestrator itself from that trusted checkout. All orchestrator Git commands discard inherited GIT_* overrides and ignore replacement objects, legacy graft files, and global or system Git configuration. Identity, tree and ancestry checks use the actual stored objects. Validation stages receive the same Git restrictions in their credential-free environment. Git and image/package downloads require network access when not already cached. The suite never registers a runner and needs no fleet credential. Existing installer/systemd/recovery fixtures use fake services and isolated filesystem roots. Installer fixtures that mutate Git history or reviewed files use their own repository copies, preserving the candidate checkout and its Git metadata. They do not prove deployment on a real prepared host.

Installer fixture preparation seeds its three hard-pinned historical engine inputs before installing synthetic Git transport: pre-health af9c0c13cd12866ce75dd6c43a4cda01915507e1, pre-telemetry 8bd27a2e7451806a1e37f79a15eeb7c677a12c3e, and pre-adapter 003de44f7e27d7bfe5bb753098efb6eb5f9712f1. It prefers available original-source objects; a missing object requires a canonical pinned fetch during that admitted preparation phase. It verifies each requested commit without changing the selected HEAD. Subsequent installed synthetic-engine fetches use only the prepared local fixture repository, retaining canonical origin metadata. There is no runtime network fallback. These historical inputs test compatibility refusals and removal; they are not rollout engine selections or evidence of native image acceptance.

Complete engine command

Replace all uppercase values with full available commit SHAs or local paths. Choose the PR's exact head and final base. Use a new output directory outside every Git worktree and repository, including source, policy and the executing validator checkout. The harness resolves symlinks and checks Git locations without inherited Git overrides. An unsafe or indeterminate location fails before creating directories, reports or logs, or executing candidates. Run as root only on the disposable execution machine.

sudo python3 /trusted/ci-fleet/scripts/local_validate.py \
  --repository /work/ci-fleet \
  --source SOURCE_SHA --base BASE_SHA \
  --policy POLICY_SHA --main MAIN_SHA \
  --pr-title 'ci: replace hosted validation with local testing' \
  --disposable-host --timeout 10800 --output /results/unique-run

The default profile is full. --profile quick omits container builds and cleanup end-to-end checks and explicitly records that expected skip. A quick success is never full evidence. The engine's lower-level scripts/validate.sh remains useful for development, but it omits workflow policy and other checks. It cannot claim complete validation by itself.

For a root commit or an initial push with no prior integrated state, either repository kind accepts --base 0000000000000000000000000000000000000000. Preserve that exact initial-push value even when imported history has parents. Do not substitute the source or its parent. All other commit identities must name real full commit SHAs. The report retains the zero base and records prior_integrated_state: false. Commit policy runs without a range base, and secret scanning covers every commit reachable from the source and original reviewed head. A real base retains its ancestry and range checks.

For a new release add --tag v0.2.0 --tag-created. Never pass --tag-created for a tag update. A release validates conventional messages and committed blobs back through the previous stable release, or all history for an initial release. Stable tags must be reachable from the explicitly supplied main commit; prereleases must be branch-local. The tag validator also enforces the version bump. Keep release review and production approval separate from these tests.

The runner creates independent source and policy seeds, excludes working-tree edits, resets the main reference to the supplied identity, and uses a unique private temporary directory under the operator's protected home for engine runs, or under root-owned /var/lib/ci-validation-work for unprivileged configuration stages. /run is often mounted noexec, so candidate executables do not live there. The latter receive their own writable source and scratch copies; trusted policy copies are read-only mounts inside the job. Only its source, home and scratch paths are writable. Seeds, completion receipts and reports remain root-owned. Permission fixtures use a separate uniquely named directory under /tmp with read and traversal permission for their deliberate UID changes. Each fixture controls access to its own child directory. Both temporary roots are removed on exit. It strips inherited credentials and Git/Python configuration from check environments. Each stage gets fresh source and policy checkouts from the verified seeds. In-memory manifests compare filesystem bytes, types, modes, symlink targets and Git metadata before recording success, including the index and config. Changing either checkout or a seed fails the stage, even if a Git index flag would hide the edit from git status. Generated regular Python caches are allowed within a stage and discarded before the next one. The commit validator, secret scanner and check plan come from the pinned policy. Failure diagnostics identify up to three changed relative paths and whether their type, mode or contents changed. They never include file contents or symlink targets.

Checkouts start from empty Git repositories. Source fetches include only the declared source, reviewed head, nonzero base and main commits with their complete ancestry. Policy fetches include only the pinned policy commit and its ancestors. Neither checkout imports unrelated branches or tags. Release runs select the highest valid prior stable tag reachable from source and the highest reachable from declared main, excluding the candidate tag. The report freezes each selected tag's name, object ID and peeled commit in release_tags. Fresh source stages fetch those exact objects and refs, preserving annotated tags and release-bump policy. Policy stages receive no tags.

These checks prevent persistent substitution between stages. They do not sandbox candidate code with root or Docker access, detect a file changed and restored within one stage, or protect the orchestrator and system tools from a hostile root process. Source programs require independent review and supervised execution on the disposable machine. Keep the signing authority elsewhere. Trusted inline Python drivers use isolated mode so candidate modules cannot replace the standard-library imports that dispatch commit and history checks.

Each stage has a log, timestamps, result and SHA-256 log digest in result.json. The report records commit, tree, base, main, suite, profile, policy revision, metadata, tool versions and expected skips. Preserve the entire output directory outside Git. A changed source, base, title, tag or policy requires a new run. Nonzero exit means failure or incomplete execution; no report also means incomplete execution. Never infer success from the last visible test line.

The CLI defaults to 1,800 seconds per stage. The complete engine command above explicitly sets 10,800 seconds to cover the measured fixture corpus. The shared scripts/local_validation_executor.py creates a unique system-manager transient service per stage. RuntimeMaxSec enforces its deadline; KillMode=control-group includes descendants that detach or double-fork. ExitType=cgroup keeps the service alive through descendant cleanup. The client asks the manager to stop the unit when its recorded main process exits; it reads manager state, never walks or signals a process tree. A shell exiting on TERM cannot cut short its child's resource-cleanup helpers. Systemd sends TERM, allows 60 seconds for cleanup, and uses KILL if necessary. A leader exiting zero cannot hide forced cleanup. Cancellation stops only that invocation's unit and waits for cleanup. An early cancellation retries the stop request until the new unit is visible. Cancellation before service startup may prevent the completion hook from running. It still reports cancellation after the empty-cgroup check and never accepts a result; successful validation always requires the completion receipt.

The launcher loads the protected installed executor and requires its exact revision to match --policy, or an independently selected --executor-revision when configuration policy intentionally retains an older baseline. The report records that executor revision. An unrelated sibling helper cannot execute stages, and a mismatched installation fails before launching any service. The executor reads a root-protected ExecStopPost receipt and verifies the service cgroup is empty before recording success or starting another stage. Missing receipts, unsuccessful service results, forced termination and incomplete cleanup fail validation. There is no /proc walk, pidfd table or child-subreaper in the executor. Ordinary configuration stages cannot write the trusted policy, completion receipt or report, and do not receive a manager bus or Docker socket.

If the launcher is killed, PID 1 still enforces the stage deadline, but the run has no accepted completion report. A lost worker invalidates an unfinished run; rebuild and rerun against the same pinned inputs. Unkillable kernel tasks, manager failure or uncertain cleanup require discarding the execution worker. Systemd does not contain hostile root or clean external Docker-daemon resources. Privileged engine fixtures still require independent review and disposable-VM execution. Use run-scoped Docker cleanup and never global Docker prune. Failed final daemon queries or removals fail the container stage; build cache can remain on the disposable daemon.

Keep one result directory per run. Export that directory once through an authenticated operator channel to durable storage outside the worker. Verify result.json and every recorded log digest at the destination before destroying the worker. Review and signing use that same bundle. GitHub artifact uploads and an additional storage service are not required. An incomplete transfer cannot satisfy the signer because its report or required log hashes are missing.

Hosted result artifacts

The existing engine workflow's validate job exports the entire $RUNNER_TEMP/local-validation/ output directory on success or failure as local-validation-RUN_ID-RUN_ATTEMPT. Download it from the workflow run's Artifacts section within seven days. The upload includes hidden files and fails when output is missing. The preceding step makes only that result directory readable by the runner account. Workflow permissions and trusted checks remain unchanged. The job uses the infrastructure timeout documented below.

Retain result.json and every recorded log and verify their hashes after download before disposing of the worker. Artifact archives do not preserve protected Unix ownership and modes; restore those protections through the existing operator procedure before treating a downloaded bundle as protected local evidence. An artifact upload is unsigned developer evidence. It does not replace required checks, independent review, signing or native/host acceptance. Early admission refusal may produce no output. Cancellation, job timeout or a lost worker can prevent export, and missing reports/logs remain incomplete validation. Local runs still use the operator export procedure above and need no GitHub artifact.

Configuration repositories

Use the same trusted engine orchestrator with a separate source repository:

sudo python3 /trusted/ci-fleet/scripts/local_validate.py \
  --kind configuration --repository /work/private-config \
  --source APPLIED_SHA --reviewed-head REVIEWED_HEAD_SHA \
  --base FIRST_PARENT_SHA --main APPLIED_SHA \
  --policy-repository /trusted/ci-fleet --policy REVIEWED_ENGINE_SHA \
  --output /results/config-run

This runs conventional-commit policy, trusted committed-history secret scanning, syntax checks, pinned-policy rejection tests against public fixtures, strict source validation against the base's prior configuration and rollout evidence, reference configurations and initializer positive/negative checks. The strict validator and its reference schema come from the independently pinned engine, so candidate validator changes cannot waive host policy. Source policy tests are exercised as engine/template development, while initialized configuration runs use the pinned public fixtures. It reads prior files from Git blobs, not from mutable temporary files belonging to another run. Initialized adopters must pass strict policy; public fictional fixtures remain non-strict in the engine's embedded-template tests.

A merged or squash commit must obtain its own full configuration result with its first parent as base and --reviewed-head naming the original reviewed head. The trees must match. Commit policy and secret scanning cover both histories, including a secret removed before a squash. Reviewed-head results alone do not authorize the new commit, even when its tree matches. Every integrated commit after the installed baseline needs its own result and review evidence. Reconcile and repeat review when a base change changes the proposed tree.

With the zero-base sentinel, transition validation uses empty prior controllers and no prior rollout or approval evidence. A root or initial-push configuration cannot supply its own prior capability or activation evidence to bypass staging. A developer report with a zero base does not authorize an ordinary applied commit. Host verification still requires that commit's real first-parent base and the existing signed-evidence checks.

Coverage and hosted adapter

Previously workflow-only check	Complete local implementation
PR title and proposed conventional messages	Pinned validate_commits.py
Tag immutability, SemVer, main ancestry, release range and bump	Explicit tag metadata and pinned validator
Secret scan of every proposed committed blob	Pinned scan_committed_secrets.py, including removed intermediate secrets
ShellCheck and documentation links	Local stages against the source checkout
Python, task-plan, installer, systemd, recovery and security tests	Pinned validate.sh --quick, with named tests in its log
Compose validation, image builds, APT cache reuse/image-ID parity, inert runner and cleanup outcomes	Pinned validate.sh --containers
Embedded schema, policy, multi-host and initializer checks	Pinned validate-template.sh

The engine workflow calls the full local entry point and preserves the Build without registering a runner check identity.

Both engine jobs select the runner label in repository variable CI_FLEET_VALIDATION_RUNNER, defaulting to ubuntu-latest when unset. To avoid hosted minutes, set it to a unique label on an operator-approved, disposable self-hosted Linux worker before enabling or dispatching validation. Run only reviewed source on that worker. Use one-job runners, remove their registrations afterward, and keep validation paused if no approved worker is available. The worker still needs the tools, systemd and isolation described above. This selection does not change required check names. The complete infrastructure job has a 240-minute ceiling and 10,800-second stage budgets. A full fixture attempt reached its 7,200-second limit after the Python, tester and controller-installer groups passed. The remaining deployer-installer section took 24 minutes when measured separately on the four-CPU disposable worker. The larger bounds cover the complete fixture corpus and subsequent image builds; they do not establish a passing run or reliability. The convention job and ordinary task jobs keep their five-minute limits. The APT acceptance build client stays alive until fixture cleanup; it has no separate 30-minute expiry that can interrupt the managed image builds.

Its existing trusted-main commit and secret policy guards remain separate during staging. Manual branch dispatches use the merge base with freshly fetched main for both commit-message validation and scanning every proposed committed tree. Manual tags use the current-main release baseline. A first release without a prior baseline still scans all reachable history. Removed secrets within the proposed range remain failures; unrelated integrated history does not replace that range.

Engine PR runs use a per-PR concurrency group, and both jobs skip events for closed PRs. Open PR title and target-base edits still rerun validation without cancelling main push validation. Hosted PR jobs execute the proposed suite from the checked-out revision, as before; trusted local acceptance instead pins an independently reviewed suite outside the candidate. Protection migration remains separately gated. The embedded template convenience job is named hosted-validation and is default-off. Set repository variable CI_FLEET_HOSTED_VALIDATION=true and set CI_FLEET_VALIDATOR_REVISION to an independently reviewed full supporting engine commit to enable it. Its PR runs use the target base SHA as both base and main, and validate the PR title passed as a quoted argument. For PR events, the optional job requires an open PR. Title or target-base edits rerun validation while the PR is open. Closed PR edits skip the job. Concurrency groups include the event name and PR number or ref, so a merged PR edit reporting the main ref cannot cancel main push validation. Push runs use the source SHA as main and retain the event's base, including the zero sentinel for an initial push. The standalone template must adopt the same behavior under #32.

The optional job never emits the old validate check context, including when disabled. A skipped optional job must not satisfy a required validate check. Keep existing protections until a deliberate reviewed migration reconciles that required context. Enabling the convenience job does not add a required check or a host requirement. Disabling the variable, removing the workflow, Actions outages and quota blocks must leave local execution complete.

Cutover gates

1. Retain a full exact-identity engine run on a disposable Docker host with Actions unavailable, plus successful complete configuration-consumer runs. Retain inspectable reports and logs and compare them with existing coverage.
2. Finish template #32, including its standalone-only tests and optional workflow. Publish a template release only through its existing reviewed release gate.
3. Stage an engine, executing manager and checkpoint manager advertising configuration_local_validation. Exercise candidate rejection, rollback and last-known-good recovery with Actions unavailable. Provision local approval policy only after compatible recovery is verified.
4. Deliberately reconcile required checks and auto-merge under engine issue #114. An Actions check cannot require an unavailable optional workflow. Preserve independent review and unrelated protection. Do not permanently waive a failed check.
5. Adopt the template and private configuration changes separately. Record the cutoff, baseline, evidence signing authority, policy and tested rollback or reinstall procedure. Restore the previous reviewed policy and workflow only through an authorized migration if cutover fails.

No billing, repository setting, credential, release, private engine pin or live host change follows automatically from repository implementation.