Skip to content

Commit 67beabb

Browse files
authored
feat(ci): add trusted local validation without Actions
Add complete local engine and configuration validation with pinned trusted checks, isolated stages, descendant cleanup and exact-source reports. Add independently signed policy-v2 evidence while retaining existing required checks and normal project Actions. Validation: exact head a8552da passed all eight standalone stages with no skips, all 68 restoration/verification checks, independent archive readback and required hosted CI. Signed host adoption, recovery and protection migration remain separate cutover work. Refs #135 Template follow-up: RandomDevelopment/ci-fleet-config-template#32
1 parent b18e6a5 commit 67beabb

37 files changed

Lines changed: 3669 additions & 203 deletions

‎.github/workflows/validate.yml‎

Lines changed: 18 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -167,53 +167,25 @@ jobs:
167167
- name: Install static analysis tools
168168
run: sudo apt-get update && sudo apt-get install -y shellcheck python3-yaml
169169

170-
- name: Check shell scripts
171-
run: shellcheck scripts/*.sh examples/project/scripts/ci/*.sh templates/config-repository/scripts/*.sh
172-
173-
- name: Check documentation links
174-
run: python3 scripts/check_doc_links.py
175-
176-
- name: Build and inspect inert prototype
177-
run: scripts/validate.sh
178-
179-
- name: Validate public configuration template
170+
- name: Run the complete local suite
171+
env:
172+
SOURCE: ${{ github.event.pull_request.head.sha || github.sha }}
173+
BASE: ${{ github.event.pull_request.base.sha || github.event.before }}
174+
PR_TITLE: ${{ github.event.pull_request.title }}
175+
EVENT_NAME: ${{ github.event_name }}
176+
REF_TYPE: ${{ github.ref_type }}
177+
TAG_NAME: ${{ github.ref_name }}
180178
run: |
181-
cmp -s scripts/scan_committed_secrets.py templates/config-repository/scripts/scan_committed_secrets.py
182-
python3 templates/config-repository/scripts/test_policy.py
183-
python3 templates/config-repository/scripts/scan_committed_secrets.py
184-
templates/config-repository/scripts/validate.sh
185-
templates/config-repository/scripts/validate.sh --config templates/config-repository/examples/multi-host/fleet.json
186-
temporary_directory="$(mktemp -d)"
187-
if templates/config-repository/scripts/init.sh \
188-
--organization test-company \
189-
--project test-app \
190-
--output "${temporary_directory}/missing-engine.json"; then
191-
echo 'initializer accepted a missing engine revision' >&2
192-
exit 1
193-
fi
194-
test ! -e "${temporary_directory}/missing-engine.json"
195-
if templates/config-repository/scripts/init.sh \
196-
--organization test-company \
197-
--project test-app \
198-
--registry BAD \
199-
--engine-ref "$(git rev-parse HEAD)" \
200-
--architecture amd64 \
201-
--controller-image-digest "sha256:$(printf '3%.0s' {1..64})" \
202-
--runner-image-digest "sha256:$(printf '4%.0s' {1..64})" \
203-
--output "${temporary_directory}/invalid.json"; then
204-
echo 'initializer accepted an invalid registry' >&2
205-
exit 1
206-
fi
207-
test ! -e "${temporary_directory}/invalid.json"
208-
templates/config-repository/scripts/init.sh \
209-
--organization test-company \
210-
--project test-app \
211-
--engine-ref "$(git rev-parse HEAD)" \
212-
--architecture amd64 \
213-
--controller-image-digest "sha256:$(printf '3%.0s' {1..64})" \
214-
--runner-image-digest "sha256:$(printf '4%.0s' {1..64})" \
215-
--output "${temporary_directory}/fleet.json"
216-
test "$(stat -c '%a' "${temporary_directory}/fleet.json")" = 644
179+
main=$(git rev-parse origin/main)
180+
if [[ -z "$BASE" || "$BASE" =~ ^0+$ ]]; then BASE=$(git merge-base "$SOURCE" "$main"); fi
181+
args=()
182+
if [[ "$EVENT_NAME" == pull_request ]]; then args+=(--pr-title "$PR_TITLE"); fi
183+
if [[ "$REF_TYPE" == tag ]]; then args+=(--tag "$TAG_NAME" --tag-created); fi
184+
# The convention job independently rejects moved tags. The trusted
185+
# main scanner above remains a separate guard during migration.
186+
sudo python3 scripts/local_validate.py --repository "$PWD" \
187+
--source "$SOURCE" --base "$BASE" --policy "$(git rev-parse HEAD)" --main "$main" \
188+
--disposable-host --output "$RUNNER_TEMP/local-validation" "${args[@]}"
217189
218190
- name: Require successful convention validation
219191
env:

‎AGENTS.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,3 +82,15 @@ Keep changes small and reversible. Existing project CI must continue operating u
8282
- Identify the source engine issue/PR, full reviewed commit, gap, affected template files, acceptance checks, capability staging, migration requirements, and documentation impact. Mark unmerged features pending and update the follow-up with the full merged commit when available.
8383
- Link the template issue in the engine PR description and completion report, or explain why no template change is needed. If issue lookup, creation, or update fails, report the missing tracking as outstanding rollout work; do not claim template synchronization.
8484
- A tracked issue is not a completed template change. Engine availability, a published template release, adopter engine promotion, and live-host acceptance require separate evidence and authorization. This workflow grants no template merge, release, private configuration change, or host activation authority. Keep ordinary CI read-only; no new automation credentials are required.
85+
86+
## Local validation and evidence
87+
88+
- Use `scripts/local_validate.py` for complete exact-source/base validation.
89+
Follow `docs/LOCAL-VALIDATION.md` for tools, isolation, profiles and logs.
90+
- A quick run or `scripts/validate.sh` alone is incomplete full-suite evidence.
91+
Missing tools and unexpected skips must fail. Privileged fixtures belong on
92+
disposable isolated execution machines, never active fleet hosts.
93+
- Trusted local acceptance uses policy-v2 signed evidence from an independent
94+
authority and a pinned validator. Keep signing keys off execution machines.
95+
- Preserve current required checks until coverage, recovery and a deliberate
96+
policy migration pass. Optional hosted convenience never creates a host gate.

‎README.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,12 @@ Use one shared pool of disposable CI workers across multiple trusted private rep
1414

1515
> **Status:** Controlled migration. The isolated one-job pilot passed on 2026-07-15. Reviewed schema-v3 desired state now governs managed ordinary-CI controller lifecycle. Tester and deployer components remain under review, and production readiness remains gated. See [accepted design decisions](docs/DESIGN-DECISIONS.md).
1616
17+
Run the complete engine and embedded-configuration suites without Actions through
18+
[local validation](docs/LOCAL-VALIDATION.md). Signed local configuration evidence
19+
is opt-in; existing host and required-check policies need a reviewed migration.
20+
The [operator walkthrough](docs/LOCAL-VALIDATION-RUNBOOK.md) gives the setup,
21+
execution, signing and acceptance sequence.
22+
1723
## What problem does this solve?
1824

1925
Self-hosted CI often grows one project at a time:

‎docs/COMPLIANCE-CHECKLIST.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,3 +60,13 @@ A project MUST complete this checklist before its existing required CI is moved
6060
- [ ] Existing required checks remained available during validation.
6161
- [ ] Rollback instructions were tested.
6262
- [ ] Project-specific runners were retained through the observation period.
63+
64+
## Engine and configuration validation migration
65+
66+
The [local validation contract](LOCAL-VALIDATION.md) governs replacement of
67+
engine and configuration hosted validation. It leaves application fleet jobs
68+
and their five-minute task contract unchanged. Preserve independent review,
69+
current required checks and recovery until complete local parity and deliberate
70+
policy cutover pass. Optional hosted convenience is separate from a policy
71+
requiring hosted success. Record template adoption and live-host acceptance
72+
separately from engine implementation.

‎docs/CONFIGURATION-APPROVAL.md‎

Lines changed: 112 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -24,14 +24,16 @@ Rollback and uninstall preserve both. Only a trusted root operator can remove
2424
the gate; it does not defend against a compromised host root or trusted engine.
2525

2626
The policy pins the configuration repository's numeric GitHub ID, a simple
27-
branch name such as `main`, the validation workflow's numeric ID, required job
28-
names, and allowed reviewer numeric IDs. These are trust anchors, not application
27+
branch name such as `main`, allowed reviewer numeric IDs and validation evidence
28+
requirements. Version 1 pins the workflow ID and required job names. Version 2
29+
pins signed local evidence and can additionally require hosted results. These are trust anchors, not application
2930
allowlists or host inventory. Account renames do not substitute a different
3031
reviewer identity. The verifier checks the requested repository against its
3132
numeric ID and default branch, and checks the local Git tree against GitHub.
3233

33-
The existing host-side App needs read-only Contents, Pull requests, and Actions
34-
permissions on the configuration repository. It receives no approval or write
34+
The existing host-side App needs read-only Contents and Pull requests
35+
permissions on the configuration repository. Policies requiring hosted results
36+
also need read-only Actions permission. It receives no approval or write
3537
authority. Long-lived credentials remain on the controller, outside ordinary CI.
3638
Tokens travel to the verifier through stdin, never command arguments, policy,
3739
or logs. Errors do not print API responses, policy values, or credentials.
@@ -63,7 +65,9 @@ adapter's resume action, the trusted verifier requires all of the following:
6365
also accepts verified clean review evidence from an explicitly trusted bot.
6466
The review must follow any base-branch retargeting. Unavailable or incomplete
6567
PR base history rejects the candidate.
66-
4. The latest push run of the pinned workflow on the applied SHA completed
68+
4. The policy's validation evidence must pass. Version 2 requires the signed
69+
local result described below. Policies with hosted requirements also require
70+
the latest push run of the pinned workflow on the applied SHA to complete
6771
successfully for the trusted repository and branch. Each required named job
6872
also completed successfully in that exact run attempt. Skipped, failed,
6973
cancelled, queued, and missing evidence is rejected.
@@ -174,3 +178,106 @@ Deterministic tests cover review/check rejection, tree identity, API failures,
174178
pagination, policy permissions, direct installer modes, remote reconciliation,
175179
resume, and rollback. Real permissions, reviewer provisioning, systemd execution,
176180
and unattended bypass rejection require separately authorized live verification.
181+
182+
## Signed local validation
183+
184+
The [operator walkthrough](LOCAL-VALIDATION-RUNBOOK.md) covers prerequisite setup,
185+
key/policy bootstrap, independently verified transfer, signing, protected evidence
186+
installation and read-only host acceptance checks.
187+
188+
Policy version 1 retains its hosted-workflow requirement. Policy version 2
189+
requires signed full local validation and omits Actions calls when both
190+
`workflow_id` and `required_jobs` are absent. Adding both fields makes hosted
191+
success an explicit additional operator requirement. An optional convenience
192+
workflow alone never adds this requirement. Use the
193+
[local policy example](../host/config-review-policy-local.json.example) only
194+
after replacing every fictional identity, revision and the public test key.
195+
The example key is a published test vector and must never authorize a host.
196+
197+
Version 2 requires `configuration_local_validation` on the executing engine,
198+
candidate engine and checkpoint manager. Existing capability checks enforce
199+
this before activation and rollback restoration. Stage compatible managers,
200+
verify recovery, then provision the root-owned policy in a separate authorized
201+
host change. Policy and marker ownership rules remain unchanged. Downgrade
202+
requires a deliberate compatible policy migration, never ignoring an unknown
203+
policy or silently falling back to hosted results.
204+
205+
The local policy pins the suite name, validator revision, exact required stage
206+
names, Ed25519 public key, protected evidence directory and maximum age. Its
207+
canonical JSON SHA-256 binds every policy field into the signed evidence. Key
208+
rotation, reviewer changes, a different suite or any other policy change
209+
invalidates earlier evidence. A result must name the exact integrated commit,
210+
its tree and first parent, the original reviewed history, the PR number and current reviewed head. All stages
211+
must pass with no skips. Future timestamps and evidence older than the policy
212+
limit reject acceptance. The installed-baseline ancestry rules continue to
213+
reject rollback replay; rechecking the same installed commit for drift remains
214+
idempotent and still requires fresh, valid evidence.
215+
216+
The signer is a separately trusted operator authority. The signature attests
217+
that this authority independently verified execution of the pinned full suite
218+
with the recorded inputs and logs. It is not hardware attestation. The signer
219+
must obtain results through its own supervised execution or an authenticated
220+
execution channel outside candidate control. A candidate upload, success file,
221+
PR comment or an unverified copy of a report is insufficient. The signing helper
222+
checks report structure and log hashes; it cannot establish how an arbitrary
223+
uploaded report was produced. Never expose it as a blind upload-signing service.
224+
225+
Keep the private signing key on a separate trusted machine or isolated signing
226+
service. Never place it in a checkout, workflow, test environment, Docker host,
227+
container or ordinary job. Docker access is host-root-equivalent. A second Unix
228+
user on the execution machine does not protect the signer. Use a fresh disposable
229+
execution machine without controller credentials, then independently verify and
230+
transfer its results through the authorized operator channel.
231+
232+
After that independent verification, the trusted operator can sign on the
233+
separate signing machine using the reviewed helper:
234+
235+
```bash
236+
python3 /trusted/ci-fleet/scripts/attest_local_validation.py \
237+
--policy /etc/ci-fleet/config-review-policy.json \
238+
--result /verified/config-run --key /protected/local-validation.key \
239+
--reviewed-head REVIEWED_HEAD_SHA --pull-request 42 \
240+
--output /verified/APPLIED_SHA.json
241+
```
242+
243+
The signing machine needs a protected copy of the policy and activation marker.
244+
The helper refuses quick or incomplete results, unexpected coverage, changed
245+
logs and the wrong validator revision. It also rejects already unusable report
246+
timestamps before signing or creating output. Both timestamps must be integers,
247+
excluding booleans, with a positive start no later than the finish and a finish
248+
no later than the signer's current time. Age from the original start must not
249+
exceed the protected policy's `local_validation.max_age_seconds`. Signing
250+
preserves the recorded timestamps and does not extend their expiry. A candidate
251+
report alone never establishes execution provenance.
252+
Before publishing evidence, the helper verifies the generated signature with
253+
the protected policy's public key. A wrong private key fails without creating
254+
the exclusive output file; correct the key or reviewed rotation policy first.
255+
The pull-request number must be positive. Zero and negative values are rejected
256+
before the helper reads policy, uses the signing key or creates output.
257+
Before signing, it rejects a report, log,
258+
or encoded evidence envelope over the verifier's 1 MiB file limit.
259+
The signer must also check source, tree,
260+
base, metadata and tool versions against its independently observed run. The
261+
host verifier repeats context, freshness, policy, signature and coverage checks.
262+
263+
Provision `APPLIED_SHA.json` and the stage logs as root-owned regular files with
264+
mode `0600` under the fixed policy directory. Put the logs at
265+
`APPLIED_SHA.logs/CHECK_NAME.log`. Parent directories must be root-owned and
266+
not group- or other-writable. Symlinks, hard-linked files, unsafe permissions,
267+
invalid JSON, missing logs and changed log digests reject acceptance. Each file
268+
has a 1 MiB limit. Keep larger diagnostic archives separately; reduce noisy
269+
output deliberately rather than truncating accepted logs after signing.
270+
Neither evidence nor keys belong in Git or runner configuration. The verifier
271+
uses bounded protected reads, invokes OpenSSL without a shell and never prints
272+
API bodies or evidence contents.
273+
274+
GitHub remains required for repository identity, commits, PR association,
275+
ancestry comparisons, independent reviews, base-change history and optional
276+
Codex evidence. App token issuance and Git fetch also remain GitHub dependencies.
277+
Version 2 removes the Actions API dependency only when hosted fields are absent.
278+
GitHub review/API outages still reject new application. Trusted last-known-good
279+
recovery uses the existing protected recovery record and needs neither Actions
280+
nor a newly signed local result. It preserves the policy and credentials.
281+
282+
See [local validation and cutover](LOCAL-VALIDATION.md) for execution commands,
283+
coverage, merge/squash evidence, remaining template work and migration gates.

‎docs/CONTRIBUTING.md‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -206,7 +206,8 @@ credentials; ordinary CI keeps read-only permissions. See the
206206
207207
A version is released (tagged on `main`) only when:
208208
209-
- the tagged commit passes all CI checks;
209+
- the tagged commit passes the complete [local suite](LOCAL-VALIDATION.md), and
210+
all established required hosted checks until their deliberate migration;
210211
- the change set is reviewed and the SemVer bump matches the Conventional
211212
Commits classification;
212213
- an operator has confirmed the live pilot evidence for any engine rollout
@@ -252,3 +253,10 @@ python3 scripts/test_validate_commits.py
252253
python3 scripts/validate_commits.py --message - <<< 'feat: local example'
253254
python3 scripts/validate_commits.py --version 0.1.0
254255
```
256+
257+
The complete validation command and evidence contract are documented in
258+
[local validation](LOCAL-VALIDATION.md). Local execution needs no Actions event,
259+
`gh`, hosted allowance or Actions API. The current required check remains active
260+
while parity and rollback are pending. Issue #114 must reconcile auto-merge with
261+
the selected evidence policy before a required check is removed. A locally
262+
passing subset never permits a release or bypasses review.

‎docs/DESIGN-DECISIONS.md‎

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -225,9 +225,26 @@ Legacy checkpoint image IDs do not make their mutable build inputs reproducible.
225225

226226
The remaining risks are incorrect operator sequencing and inaccurate host evidence. Issue #36 retains the operational bootstrap, repeated-build and private image-ID proof. Remove this transitional exception after participating hosts have adopted image enforcement and the retained pre-enforcement rollback paths are retired through reviewed policy.
227227

228+
## Local engine and configuration validation
229+
230+
Status: implementation proposed under issue #135. Operational cutover pending.
231+
232+
The [complete local suite](LOCAL-VALIDATION.md) removes Actions scheduling from
233+
test execution. Policy-v2 [signed local evidence](CONFIGURATION-APPROVAL.md#signed-local-validation)
234+
can replace Actions results while retaining independent GitHub review and
235+
ancestry checks. An independently trusted signer attests supervised execution;
236+
this is not hardware attestation or an automatic candidate-upload signing path.
237+
Signing authority must remain outside the Docker execution boundary.
238+
239+
Existing root-owned policies and required-check settings retain their behavior.
240+
Full isolated Docker proof, compatible manager/checkpoint recovery, template
241+
adoption and reviewed policy cutover remain required. No host, credential,
242+
branch-protection, auto-merge, engine promotion or production change is authorized
243+
by this implementation.
244+
228245
## Engine integration validation budget
229246

230-
Status: proposed in PR #136.
247+
Status: accepted in merged PR #136.
231248

232249
The read-only hosted engine integration job has a 45-minute ceiling. Repository
233250
regressions measured about 24 minutes before image acceptance; optional APT cache

0 commit comments

Comments
 (0)