Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/validate.yml
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ jobs:
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
repository: RandomDevelopment/ci-fleet
ref: 689c7c00deb0eebcb25f2284334a9f7168a25575
ref: c51844cf05f7a9f647322f4b7cbbb6379f40ad5d
path: .ci-fleet-executor
persist-credentials: false
fetch-depth: 1
Expand Down
8 changes: 8 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -176,3 +176,11 @@ through a reviewed ownership map. Never replace the private suite with upstream
local_validate.py or policy tests. Repository suites precede installation rehearsal
because installed fleet-role markers block validation. A modest rehearsal cannot
substitute for actual target capacity and resource admission.

For socket-proxy upload memory recovery, follow
[the ordered guidance](docs/UPDATING.md#socket-proxy-upload-memory-and-recovery).
Keep spools disk-backed and preserve the 128 MiB memory limit, 1 GiB body cap
and two upload slots. Accept exact source and complete immutable image evidence
separately from authorized host adoption and independent live availability.
Guidance imports must preserve runtime/image pins and uncertainty guards;
source acceptance cannot release a held application retry.
6 changes: 6 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -359,6 +359,12 @@ follow [manager adoption and recovery outcomes](docs/UPDATING.md#engine-manager-
An unchanged applied SHA or a successful systemd unit does not prove health or
successful reconciliation.

For Docker build or image-load upload memory incidents, follow the
[socket-proxy recovery sequence](docs/UPDATING.md#socket-proxy-upload-memory-and-recovery).
It requires disk-backed spools with the unchanged 128 MiB memory limit,
1 GiB body cap and two upload slots. Accepted source, complete immutable image
artifacts and independent live availability have separate acceptance gates.

### Optional host-enforced configuration approval

A compatible installed engine advertising `configuration_approval_gate` can
Expand Down
19 changes: 13 additions & 6 deletions docs/LOCAL-VALIDATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,9 +30,13 @@ protected workspace parent. It adds no daemon, timer or unique fleet state.
Use a Linux VM or remote prepared worker from macOS or Windows. Shared executor support merged in core [PR #144](https://github.com/RandomDevelopment/ci-fleet/pull/144)
at `689c7c00deb0eebcb25f2284334a9f7168a25575`. The ordinary-CI
configuration-policy baseline is `840fb9be8a1c69404a5eebcf17855eff13c6c928`.
The executor remains independently pinned to the reviewed
`689c7c00deb0eebcb25f2284334a9f7168a25575` implementation. Engine merge,
template acceptance and host activation remain separate.
The executor is independently pinned to merged core
[PR #165](https://github.com/RandomDevelopment/ci-fleet/pull/165) at
`c51844cf05f7a9f647322f4b7cbbb6379f40ad5d`. Its completion hook avoids site
initialization and unnecessary imports while preserving strict receipt checks,
deadlines and cgroup cleanup. The optional workflow checks out this same executor;
adopters must select a reviewed suite pin requiring that revision.
Engine merge, template acceptance and host activation remain separate.
Executor and policy pins are independent inputs. Pinning an executor
alone does not adopt the managed-image contract or activate a runtime.

Expand Down Expand Up @@ -278,7 +282,10 @@ of these seeds in memory, including digests of file bytes, entry types and modes
symlink targets and `.git` metadata. Git status or index checks alone cannot detect all
changes, such as assume-unchanged files or edits to `.git/config`.

Initial seeds start as empty Git repositories. Source fetches use
Initial seeds start as empty Git repositories with `git init --template=`.
Optional initialization templates, such as sample hooks, are not copied into
the seeds or their stage copies. Committed files and all generated Git metadata
remain covered by the same manifests and tamper checks. Source fetches use
`--no-tags` and explicit source, original reviewed head, main and nonzero base
SHAs with their complete ancestry. Engine-policy and trusted-suite fetches
include only their selected commit and its ancestry. Checkouts inherit no
Expand Down Expand Up @@ -443,8 +450,8 @@ The core local-evidence contract merged through
passed all eight stages outside Actions on reviewed source and policy
`a8552da5c902c54c0d78b19e6f093e2e96482f62`, with `expected_skips: []`.
That engine result does not replace this complete standalone suite.
The development ordinary-CI compatibility baseline is
`689c7c00deb0eebcb25f2284334a9f7168a25575`. The historical Git/output helpers retain their reviewed implementation provenance at
The standalone executor pin is
`c51844cf05f7a9f647322f4b7cbbb6379f40ad5d`. The historical Git/output helpers retain their reviewed implementation provenance at
`96f277b3a75f3d2bef3c04ceb7b3c8778b8998c2`.
Core availability and functional suite results establish no template release,
private engine promotion or signed operational acceptance.
Expand Down
52 changes: 52 additions & 0 deletions docs/UPDATING.md
Original file line number Diff line number Diff line change
Expand Up @@ -572,6 +572,58 @@ and retry behavior remain unchanged. These observations authorize no cleanup
or gate bypass. Engine merge, template publication, private engine promotion
and separately authorized host acceptance are distinct milestones.

### Socket-proxy upload memory and recovery

Merged engine `1c685dacc109597641fd13dfae48606e76f73ee7` adds bounded
file-cache handling for Docker build and image-load uploads. Its reviewed
[host-maintenance contract](https://github.com/RandomDevelopment/ci-fleet/blob/1c685dacc109597641fd13dfae48606e76f73ee7/docs/HOST-MAINTENANCE.md#docker-storage-retention)
requires disk-backed temporary spools. The proxy synchronizes writes and
discards cached pages in 4 MiB windows while receiving uploads, then discards
consumed pages while forwarding them. Unlinking a spool alone does not release
its memory-cgroup file-cache charges. Memory-backed `tmpfs` and `ramfs` spools
are refused before forwarding. Keep the 128 MiB proxy memory limit, 1 GiB
per-body cap and two shared upload slots unchanged.

This guidance does not select that engine or change the compatibility baseline.
Apply the following order before treating an upload-memory incident as recovered:

1. Accept the exact engine source and its isolated regression evidence, including
actual cgroup memory and file-cache measurements during receiving and
forwarding. Keep earlier failed results under their original identities.
2. Accept the actual template guidance, then import its exact reviewed immutable
revision through the [unpublished-revision procedure](#import-an-unpublished-reviewed-revision).
Preserve adopter-owned validation, configuration, image pins and
`TEMPLATE_RELEASE`. Complete each repository's required suites and independent
review. These guidance changes do not stage a new runtime.
3. Separately prepare and review complete controller and runner artifacts for
the accepted merged engine and target architecture. Follow the
[managed-image procedure](MANAGED-IMAGES.md), compare original config digests
across clean repetitions, and retain complete archives, hashes and original
descriptor evidence outside Git. Temporary CI tags, an older source's proof
or one completed role cannot establish a complete target image pair. Keep
config digest C distinct from store runtime ID R and manifest bindings.
4. Only through a separately authorized adoption, stage target evidence and
image policy against the actual predecessor, then use the supported ordered
manager/runtime installation and recovery procedure. Verify current disk
admission and workload protections. Retain compatible incumbent source,
exact images and checkpoints. Preserve installer/proxy locks and uncertainty
guards. Never copy a live binary into place, bypass the proxy or clear
current-boot markers to force an upgrade.
5. Verify the actual installed source and image identities against that packet.
Independently establish sustained Docker daemon, proxy and controller
availability before releasing held application retries. A source proof,
successful build, running container, automatic proxy restart or one healthy
observation does not establish live recovery.

A failed or incomplete artifact build leaves the image gate open even when
source validation passed. Preserve its evidence and diagnose the failed
assertion before proposing a changed-source correction or another build.
An upload EOF or unavailable Docker endpoint alone does not distinguish proxy
memory-cgroup OOM, daemon failure or another transport fault. Retain scoped
incident evidence rather than inferring a cause from a timeout exit code.
Storage reclamation, capacity changes, workload retirement and application
retry authority remain separate decisions.

## Optional Docker network and storage policy

The reviewed ordinary-CI compatibility baseline is
Expand Down
4 changes: 2 additions & 2 deletions scripts/local_validate.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
ROOT = Path(__file__).resolve().parents[1]
EXECUTOR_PATH = Path("/usr/local/lib/ci-fleet-validation/executor.py")
# Shared execution tooling is independent of the configuration-policy baseline.
EXECUTOR_REVISION = "689c7c00deb0eebcb25f2284334a9f7168a25575"
EXECUTOR_REVISION = "c51844cf05f7a9f647322f4b7cbbb6379f40ad5d"
SHA = re.compile(r"[0-9a-f]{40}")
NO_PRIOR_STATE = "0" * 40
SUITE = "ci-fleet-config-standalone-v2"
Expand Down Expand Up @@ -72,7 +72,7 @@ def clone(repository: Path, revision: str, target: Path, identities: list[str])
if any(not SHA.fullmatch(value) or value == NO_PRIOR_STATE for value in identities):
raise ValueError("checkout identities must be full nonzero commit SHAs")
target.mkdir()
subprocess.run(["git", "init", "--quiet", str(target)], env=git_environment(), check=True)
subprocess.run(["git", "init", "--quiet", "--template=", str(target)], env=git_environment(), check=True)
subprocess.run(["git", "-C", str(target), "config", "remote.origin.url", str(repository)], env=git_environment(), check=True)
subprocess.run(["git", "-C", str(target), "fetch", "--quiet", "--no-tags", "--keep", "origin", *identities], env=git_environment(), check=True)
subprocess.run(["git", "-C", str(target), "checkout", "--quiet", "--detach", revision], env=git_environment(), check=True)
Expand Down
26 changes: 26 additions & 0 deletions scripts/test_local_validation.py
Original file line number Diff line number Diff line change
Expand Up @@ -508,6 +508,32 @@ def test_git_wrapper_and_clone_ignore_inherited_repository_redirection(self) ->
self.assertEqual(local.git(decoy, "rev-parse", "HEAD"), self.base)
self.assertEqual((decoy / ".git/config").read_bytes(), config)

def test_clone_omits_optional_git_init_templates(self) -> None:
template = self.root / "git-template"
(template / "hooks").mkdir(parents=True)
(template / "hooks/pre-commit.sample").write_text("unused sample hook\n")
(template / "optional-metadata").write_text("not source content\n")
target = self.root / "without-templates"
original_run = subprocess.run

def configured_init(command, **kwargs):
if command[:2] == ["git", "init"]:
command = ["git", "-c", "init.templateDir=" + str(template), *command[1:]]
return original_run(command, **kwargs)

with patch.object(local.subprocess, "run", side_effect=configured_init):
local.clone(self.repository, self.source, target, [self.source, self.base])
self.assertFalse((target / ".git/hooks").exists())
self.assertFalse((target / ".git/optional-metadata").exists())
self.assertEqual(local.git(target, "rev-parse", "HEAD"), self.source)
self.assertEqual(local.git(target, "rev-parse", self.base + "^{commit}"), self.base)
self.assertEqual((target / "marker").read_text(), "candidate\n")
manifest = local.checkout_manifest(target)
self.assertIn(".git/config", manifest)
(target / ".git/config").write_text("changed metadata\n")
with self.assertRaisesRegex(RuntimeError, "checkout changed"):
local.verify_checkout(target, manifest)

def test_stage_commands_receive_only_the_protected_git_environment(self) -> None:
def inspect(command, env, log):
self.assertEqual({key: value for key, value in env.items() if key.startswith("GIT_")}, GIT_PROTECTION)
Expand Down