[codex] ci: route builds to stable BuildKit cache owners

[kkroo]

Thinking Path

• Paperclip uses GitHub Actions to validate and publish its server and agent images.
• The image workflows execute on ephemeral ARC runners while persistent BuildKit StatefulSet pods own the reusable layer caches.
• The existing ClusterIP distributes new runner connections between both cache owners, so the same build class does not consistently return to its warm cache.
• BuildKit can also take minutes to recover a Ceph-backed cache after rescheduling, so a hard-coded pod endpoint without a bounded fallback would reduce availability.
• This pull request gives each build class a stable cache owner and retains a ready-endpoint fallback.
• The result is more predictable cache reuse without moving runtime smoke tests away from local DinD.

Linked Issues or Issue Description

No dedicated issue exists. Repeated server and agent builds can land on different BuildKit cache owners even when their inputs are unchanged, causing avoidable cache misses and longer image builds. The expected behavior is stable cache ownership with bounded failover when an ordinal is unavailable.

Related implementation PRs found during deduplication: #1185 introduced persistent BuildKit, #1193 reduced export compression cost, and #1201 is the base of this stack.

What Changed

• Select BuildKit ordinal 0 for normal server pushes and ordinal 1 for manual server builds.
• Select ordinal 1 for agent image builds while retaining local DinD for the fresh-image runtime smoke test.
• Probe the preferred endpoint with bounded TCP timeouts and fall back to the ready-endpoint Service.
• Extend workflow contract tests to prevent regression to nondeterministic routing.

Verification

• node --test scripts/__tests__/ci-cache-routing.test.mjs
• yq eval '.' .github/workflows/docker.yml
• yq eval '.' .github/workflows/docker-agent.yml
• git diff --check origin/master..HEAD
• Probed DNS and TCP port 1234 from a live ARC DinD runner to both StatefulSet ordinals and the fallback Service.

Risks

Low operational risk. A preferred ordinal can be unavailable during rescheduling or cache recovery, in which case the workflow uses only ready Service endpoints. The probe checks TCP availability rather than executing a BuildKit request, but BuildKit opens port 1234 only after worker recovery in the deployed configuration. Reverting this commit restores the shared Service endpoint.

Model Used

OpenAI Codex, GPT-5 family coding agent with repository, shell, GitHub CLI, and Kubernetes tool access. The exact deployment model identifier and context-window size are not exposed to this session.

Checklist

• I have included a thinking path that traces from project context to this change
• I have specified the model used with the available version and capability details
• I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
• I have searched GitHub for duplicate or related PRs and linked them above
• I have either linked related work or described the issue in this PR
• I have run tests locally and they pass
• I have added or updated tests where applicable
• This change does not affect the UI
• No user-facing documentation change is required
• I have considered and documented the risks above
• All Paperclip CI gates are green
• Greptile is 5/5 with no open findings
• I will address all reviewer comments before requesting merge

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

[allyblockcast]

Hey @kkroo! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• Missing section: ## Thinking Path
• Missing section: ## What Changed
• Missing section: ## Verification
• Missing section: ## Risks
• Missing section: ## Model Used
• Add the dedup-search checkbox to your PR description and check it once you have searched the GitHub PR list for similar PRs. See the PR template at .github/PULL_REQUEST_TEMPLATE.md and CONTRIBUTING.md → "Before You Start: Search First".

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 9d3565a

Critical Issues (0)

Important Issues (0)

Suggestions (0)

Strengths

• Routes each build class to a stable BuildKit ordinal while preserving a bounded healthy-Service fallback.
• Covers both server and agent workflows and keeps their Buildx remote-driver contracts explicit.
• The focused cache-routing contract test passes against the reviewed head.

Recommended Action

1. No blocking changes requested.

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 9d3565a

Critical Issues (0)

Important Issues (0)

Suggestions (0)

Strengths

• Routes each build class to a stable BuildKit ordinal while preserving a bounded healthy-Service fallback.
• Covers both server and agent workflows and keeps their Buildx remote-driver contracts explicit.
• The focused cache-routing contract test passes against the reviewed head.

Recommended Action

1. No blocking changes requested.

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 3644848

Critical Issues (0)

Important Issues (0)

Suggestions (0)

Strengths

• Routes server and agent image builds to stable StatefulSet-local BuildKit cache owners while preserving a bounded ready-Service fallback.
• Keeps the remote Buildx endpoint selection explicit and validates both workflow contracts.
• Maintains the agent runtime smoke test on its local DinD daemon rather than conflating smoke execution with remote BuildKit.

Recommended Action

1. No blocking changes requested.

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 3644848

Critical Issues (0)

Important Issues (0)

Suggestions (0)

Strengths

• Routes server and agent image builds to stable StatefulSet-local BuildKit cache owners while preserving a bounded ready-Service fallback.
• Keeps the remote Buildx endpoint selection explicit and validates both workflow contracts.
• Maintains the agent runtime smoke test on its local DinD daemon rather than conflating smoke execution with remote BuildKit.

Recommended Action

1. No blocking changes requested.