Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions docs/EXPLICIT-DRAINED-RECOVERY.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,44 @@ nonzero exit. Failed recovery must remain held; admission needs separate review
and authorization. This command does not establish active-pilot evidence,
isolation proof or permission to publish otherwise unapproved images.

## Accept an already installed recovery baseline

A supporting reviewed reconciliation checkout can accept the current installed
configuration as last-known-good without another installer transaction:

```bash
sudo /path/to/reviewed-ci-fleet/scripts/remote-reconcile.sh --accept-installed
```

Use the exact independently reviewed, merged source that supports this command.
Older managers do not support it. The runtime and installed manager do not need
to be replaced for this verification. Keep the reviewed capacities, workload
protections and automation holds in place throughout acceptance.

The command holds the installer lock and fetches only the locked installation's
full configuration identity. It requires the existing approval policy, an exact
installed check and fresh state-aware full health before replacing LKG through
the normal checked metadata writer. Authored managed-image policy is required;
image-free or staged baselines need their separate protected-checkpoint recovery
procedure and are refused by this command. Healthy or warning is accepted for active
controllers; drained and disabled controllers require matching maintenance or
warning health. The prior record remains in place during verification. Save its
protected metadata and recovery-source evidence outside Git before acceptance.

Drift, a mismatched fetch, missing or inconsistent health, and metadata write or
readback failure stop acceptance. This command never repairs configuration or
applies the older LKG, including on failure. It cannot be combined with another
reconciliation mode. Normal same-commit reconciliation, `--check-only`,
`--installed-ref` and `--no-op` retain their existing behavior and do not accept an
installed LKG. Inspect the terminal reconciliation record after any failure and
keep automation held until the actual result is established.

This provides a separate route when an older LKG has an incompatible capacity or
resource policy. It does not authorize restoration of that older state, changes
to a checkpoint, an engine upgrade, workload retirement or storage cleanup.
Template guidance and private adopter verification must follow the accepted
supporting source before live use.

## Rollout boundary

This feature requires a new reviewed supporting engine/manager pin and complete
Expand Down
51 changes: 49 additions & 2 deletions scripts/remote-reconcile.sh
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
#
# Usage:
# remote-reconcile.sh [--check-only] [--installed-ref] [--no-op]
# remote-reconcile.sh --accept-installed
set -Eeuo pipefail
set +x
export PYTHONDONTWRITEBYTECODE=1
Expand All @@ -29,6 +30,7 @@ lkg_dir=${CI_FLEET_LKG_DIR:-/var/lib/ci-fleet/last-known-good}
mode=reconcile # reconcile or check-only
no_op=false
installed_ref=false
accept_installed=false
explicit_ref=
max_attempts=${CI_FLEET_RECONCILE_MAX_ATTEMPTS:-3}
# ponytail: one-hour queue cap; align with any future installer transaction timeout.
Expand All @@ -38,15 +40,19 @@ usage() {
cat >&2 <<'EOF'
usage:
remote-reconcile.sh [--check-only] [--installed-ref] [--no-op]
remote-reconcile.sh --accept-installed

Fetches the desired-state repository at the current default-branch HEAD,
validates it, and reconciles the controller if a newer commit is available.

--ref SHA Mutating opt-in drained recovery at one full lowercase 40-hex
commit; unchanged controller/fleet policy and runtime engine only.
Incompatible with --check-only, --installed-ref and --no-op.
Incompatible with --check-only, --installed-ref, --no-op and --accept-installed.
--check-only Validate and report without reconciling.
--installed-ref Check the locked installation's commit (check-only only).
--accept-installed Verify only the locked installed commit and full health,
then accept it as LKG without applying or repairing configuration.
Incompatible with --check-only, --installed-ref, --no-op and --ref.
--no-op Log what would be done without side effects.
EOF
}
Expand All @@ -58,6 +64,9 @@ while (($#)); do
explicit_ref=$2; shift ;;
--check-only) mode=check-only ;;
--installed-ref) installed_ref=true ;;
--accept-installed)
[[ "$accept_installed" == false ]] || { echo 'ERROR: duplicate --accept-installed' >&2; exit 2; }
accept_installed=true ;;
--no-op) no_op=true ;;
-h|--help) usage; exit 0 ;;
*) echo "ERROR: unknown argument: $1" >&2; usage; exit 2 ;;
Expand All @@ -66,6 +75,7 @@ while (($#)); do
done
[[ -z "$explicit_ref" || ( "$installed_ref" == false && "$no_op" == false && "$mode" == reconcile ) ]] || { echo 'ERROR: --ref is incompatible with --installed-ref, --check-only and --no-op' >&2; exit 2; }
[[ "$installed_ref" == false || "$mode" == check-only ]] || { echo 'ERROR: --installed-ref requires --check-only' >&2; exit 2; }
[[ "$accept_installed" == false || ( "$mode" == reconcile && "$installed_ref" == false && "$no_op" == false && -z "$explicit_ref" ) ]] || { echo 'ERROR: --accept-installed cannot be combined with another mode' >&2; exit 2; }

temp_dir=$(mktemp -d) || exit 2
transaction_result=$temp_dir/transaction-result.json
Expand Down Expand Up @@ -517,6 +527,7 @@ if [[ -n "$explicit_ref" ]]; then
fetch_ref=$explicit_ref
fi
[[ "$installed_ref" == false ]] || fetch_ref=$installed_config_ref
[[ "$accept_installed" == false ]] || fetch_ref=$installed_config_ref
note "INSTALLED controller=${installed_controller} config_repo=${installed_config_repo} config_ref=${installed_config_ref}"

# Prefer host.env for token generation; fall back to rendered_env
Expand Down Expand Up @@ -554,6 +565,7 @@ while ((attempt < max_attempts)); do
done

note "RESOLVED desired_ref=${desired_commit}"
[[ "$accept_installed" == false || "$desired_commit" == "$installed_config_ref" ]] || die 'installed acceptance fetch identity mismatch'
if [[ -n "$explicit_ref" ]]; then
[[ "$desired_commit" == "$explicit_ref" ]] || die 'explicit ref fetch identity mismatch'
git -C "$temp_dir/config-repo" show "$desired_commit:fleet.json" >"$temp_dir/recovery-candidate.json" || die 'candidate unavailable'
Expand All @@ -572,6 +584,17 @@ if [[ -e "$config_review_policy" || -L "$config_review_policy" || -e "$config_re
fi
fi

if [[ "$accept_installed" == true ]]; then
git -C "$temp_dir/config-repo" show "$desired_commit:fleet.json" >"$temp_dir/acceptance-fleet.json" \
|| die 'installed acceptance configuration is unavailable'
python3 - "$temp_dir/acceptance-fleet.json" <<'PY' || die 'installed acceptance requires an activated managed-image configuration'
import json, sys
value = json.load(open(sys.argv[1], encoding="utf-8"))
images = value.get("managed_images") if isinstance(value, dict) else None
raise SystemExit(0 if isinstance(images, dict) and images else 1)
PY
fi

# Compare with installed
if [[ "$desired_commit" == "$installed_config_ref" && -z "$explicit_ref" ]]; then
# Same commit — just verify convergence
Expand All @@ -590,11 +613,35 @@ if [[ "$desired_commit" == "$installed_config_ref" && -z "$explicit_ref" ]]; the
--config-identity "$installed_config_repo" \
--ref "$installed_config_ref" \
--controller "$installed_controller" 2>"$temp_dir/drift_err"; then
health_status=healthy
if [[ "$accept_installed" == true ]]; then
save_reconcile_state 'reconciling' "$desired_commit" "$installed_config_ref" 'unknown' 'installed configuration checked; verifying health' false '' '' true \
|| die 'installed acceptance state persistence unverified'
health_status=$(run_health_check "$temp_dir/health.json")
health_desired_state=$(python3 -c 'import json, sys; print(json.load(open(sys.argv[1]))["desired_state"])' "$temp_dir/health.json" 2>/dev/null || echo unknown)
case "$installed_controller_state:$health_desired_state:$health_status" in
active:active:healthy|active:active:warning|drained:drained:maintenance|drained:drained:warning|disabled:disabled:maintenance|disabled:disabled:warning) ;;
*)
save_reconcile_state 'failed' "$desired_commit" "$installed_config_ref" "$health_status" 'installed configuration health rejected'
exit 3
;;
esac
# A failed verification must never replay an older, potentially incompatible LKG.
if ! { save_lkg "$installed_config_ref" true && lkg_readback_matches "$installed_config_ref"; }; then
save_reconcile_state 'failed' "$desired_commit" "$installed_config_ref" "$health_status" 'installed LKG persistence failed'
exit 3
fi
fi
save_reconcile_state 'converged' "$desired_commit" "$installed_config_ref" "$health_status" 'no change, converged' true '' '' true \
|| die 'installed convergence state persistence unverified'
note "CONVERGED controller=${installed_controller} config_ref=${installed_config_ref}"
save_reconcile_state 'converged' "$desired_commit" "$installed_config_ref" 'healthy' 'no change, converged' true
exit 0
fi
fi
if [[ "$accept_installed" == true ]]; then
save_reconcile_state 'failed' "$desired_commit" "$installed_config_ref" 'unknown' 'installed configuration check rejected; no repair attempted'
exit 3
fi
# Same commit + drift is still drift: check-only reports it, normal mode repairs it.
if [[ "$mode" == check-only ]]; then
save_reconcile_state 'drift' "$desired_commit" "$installed_config_ref" 'drift' 'internal drift detected'
Expand Down
Loading