fix(agent-toolchain): bake libamt_protocol, give node a writable CARGO_HOME

[eyad-hussein]

Thinking Path

• Paperclip is the open source app people use to manage AI agents for work
• Technical agents run in workspace pods built from Dockerfile.agent-toolchain, the shared image that carries the build toolchain they need to actually compile the repos they are assigned to
• Agents assigned to Blockcast/multicast cannot build cmd/caddy/sender or cmd/caddy/receiver at all: those packages link -lamt_protocol, and the library is not in the image, so every CGO_ENABLED=1 build dies at link time
• Separately, CARGO_HOME points at a root-owned directory while agents run as uid 1000, so any workspace cargo build fails on the registry cache
• Six defect rows have sat blocked on this for four days — the fixes are written and cannot be compiled, let alone tested. Every run ends in a blocked disposition naming the environment, not the code
• This pull request bakes libamt_protocol into the toolchain image and gives node a writable Cargo cache
• The benefit is that the environment stops being the blocker: agents can build and test the packages they are assigned to, and BLO-30359's five dependents become workable

Linked Issues or Issue Description

Refs BLO-30359 — Agent multicast workspace cannot build cmd/caddy/sender or produce a real PR.

Blocked behind it, and unblocked by this: BLO-30049, BLO-30253, BLO-30278, BLO-30722, BLO-29536.

The bug, in the issue-template shape:

• What happens: an agent runs CGO_ENABLED=1 go test ./cmd/caddy/sender ./cmd/caddy/receiver in its workspace and gets cannot find -lamt_protocol at link time. A workspace cargo build separately fails with Permission denied (os error 13) writing the Cargo registry cache.
• What should happen: both complete — passing or failing on assertions, not on a missing toolchain.
• Where: any agent pod on harbor.blockcast.net/paperclip-agent/paperclip-agent:*-k8s-vendored.
• Impact: six rows blocked four days, two of them critical; one gates an unplayable production live delivery service.

What Changed

Single file, Dockerfile.agent-toolchain:

• Bake libamt_protocol. New stage after the Rust block (it shells out to cargo) that shallow-fetches a pinned amt-protocol commit, runs make ffi && make install into INSTALL_PREFIX=/usr/local, then ldconfig. Mirrors the recipe multicast's own CI already uses in .github/workflows/integ-tests.yml → "Build and install amt-protocol". The repo is public, so an HTTPS fetch works and the submodule's SSH URL is not needed.
• Pin it immutably. AMT_PROTOCOL_REF defaults to ef1bf21a7b82e964dd336b8fe1f7204860c839d5 — exactly the commit multicast's own amt-protocol submodule points at, so the baked library matches what the consuming repo expects. A fail-closed guard rejects any ref that does not resolve to itself, so a branch or tag cannot be substituted.
• Two build-failing self-checks: ldconfig -p | grep -q amt_protocol and test -f /usr/local/include/amt_protocol.h.
• Writable CARGO_HOME for node at /home/node/.cargo, placed beside the existing NPM_CONFIG_PREFIX block it mirrors. RUSTUP_HOME deliberately stays at /usr/local/rustup.

Deliberately not changed: libsqlite3-dev is not added — see Risks.

Verification

Every claim below was measured in a pod running the current agent image, harbor.blockcast.net/paperclip-agent/paperclip-agent:sha-a1d2f61-k8s-vendored, not reasoned about from the Dockerfile.

Baseline — the gaps are real:

gcc:               /usr/bin/gcc          present
libamt_protocol:   0 in ldconfig -p      ABSENT
CARGO_HOME:        /usr/local/cargo      root-owned; agents are uid 1000 (node)

The AMT recipe works on that exact base:

$ make ffi && make install && ldconfig
BUILD_OK
    libamt_protocol.so (libc6,x86-64) => /usr/local/lib/libamt_protocol.so
-rw-r--r-- 1 root root     8883  /usr/local/include/amt_protocol.h
-rw-r--r-- 1 root root 22331558  /usr/local/lib/libamt_protocol.a
-rw-r--r-- 1 root root   357784  /usr/local/lib/libamt_protocol.so

The Cargo fault reproduces, and the fix resolves it (both as uid 1000):

CARGO_HOME=/usr/local/cargo   -> error: failed to open .../rand_chacha-0.3.1.crate
                                 Caused by: Permission denied (os error 13)
CARGO_HOME=/home/node/.cargo  -> exit 0, target/debug/libcgtest.rlib produced

Pinned fetch resolves against GitHub:

$ git fetch --depth 1 origin ef1bf21a7b82e964dd336b8fe1f7204860c839d5
fetch-by-SHA OK -> ef1bf21a7b82e964dd336b8fe1f7204860c839d5

Delivery path — the change actually ships:

$ scripts/container-base-tag.sh agent-toolchain <runtime>
before: toolchain-292f44fcba2b2ec9209f
after:  toolchain-d31d9975bdb2f6d75c2d

The tag moves, so docker-agent.yml rebuilds the toolchain rather than reusing the cached one, and the existing auto-bump rolls the fleet.

On tests: commitperclip flags no test files. There is no unit-test seam for a Dockerfile — the executable assertions are the two build-failing self-checks in the new stage (ldconfig -p | grep -q amt_protocol, test -f …/amt_protocol.h) plus the AMT_PROTOCOL_REF guard, all of which fail the image build rather than shipping a silent gap. That is the same discipline the file already applies with tinygo version, java -version, protoc --version and the ffmpeg -version loop. I have not retitled this refactor: — it is a fix, and mislabelling it to satisfy a check would be worse than explaining the gap. Happy to take a maintainer's call.

Risks

Low, and contained to image build time.

• Build-time network dependency on github.com/Blockcast/amt-protocol. The toolchain image already fetches from go.dev, github.com, sh.rustup.rs, download.docker.com and others, so this adds no new class of dependency. A fetch failure fails the build loudly.
• Rebuild cost: one extra cargo release build. The toolchain image is content-addressed and only rebuilt when this file changes, so this is paid once per change to it, not per commit.
• Pin drift between two repos. The baked commit and multicast's submodule pointer must be bumped together. They are identical today; the Dockerfile comment says so explicitly. The fail-closed guard prevents the sloppier failure (silently switching to a branch), not this one.
• ENV CARGO_HOME change is image-wide. Nothing after that line in the build uses cargo, and RUSTUP_HOME is untouched, so the shared toolchain still resolves — verified by the successful node-user build above.
• ⚠️ libsqlite3-dev deliberately omitted. BLO-30359 lists missing SQLite dev metadata as a fault. It is a red herring: mattn/go-sqlite3 v1.14.33 vendors the amalgamation, verified by building and running a CGO_ENABLED=1 binary importing it on the current image with no SQLite headers present (GO_SQLITE3_BUILD_OK / ok). Adding the package would be cargo cult. The risk I am accepting is that the ticket text and this PR disagree; I have posted the correction on the ticket rather than silently satisfying a wrong requirement.
• fec-raptorq is deliberately not baked. It is a submodule versioned with the multicast repo, so a per-run cargo build is correct — which the writable CARGO_HOME is what enables. Baking it would pin a repo-versioned artifact into a shared image.

Model Used

Claude Opus 5 (claude-opus-5[1m]), 1M context, extended thinking, with tool use and code execution — driving kubectl exec probes against live agent pods, gh/git, and the Paperclip API. Human-directed throughout by @eyad-hussein; all measurements were executed rather than inferred.

Checklist

• I have included a thinking path that traces from project context to this change
• I have specified the model used (with version and capability details)
• I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
• I have searched GitHub for duplicate or related PRs and linked them above — searched Blockcast/paperclip for agent-toolchain, amt-protocol, amt_protocol, libamt, CARGO_HOME across all states; no duplicate or in-flight PR touches this file for this purpose (fix(agent): unblock verified image rollout (PEN-2077) #1058 and ci(policy): fail CI when workflow content escapes its block scalar (BLO-23128) #1183 match the term but are unrelated)
• I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template
• I have run tests locally and they pass — the equivalent here is the live-image verification above; the build-time self-checks are the executable assertions
• I have added or updated tests where applicable — build-failing self-checks; see the note under Verification
• If this change affects the UI, I have included before/after screenshots — n/a, no UI change
• I have updated relevant documentation to reflect my changes — rationale is inline in the Dockerfile, where the next person to touch this stage will read it
• I have considered and documented any risks above
• All Paperclip CI gates are green — in progress
• Greptile is 5/5 with no open P2s, recommendations, or follow-ups — pending
• I will address all Greptile and reviewer comments before requesting merge

🤖 Generated with Claude Code

[allyblockcast]

🔗 Paperclip issue: BLO-30049
🔗 Paperclip issue: BLO-30278
🔗 Paperclip issue: BLO-30722
🔗 Paperclip issue: BLO-30359
🔗 Paperclip issue: BLO-29536
🔗 Paperclip issue: BLO-30253

[allyblockcast]

Hey @eyad-hussein! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• Missing section: ## Thinking Path
• Missing section: ## What Changed
• Missing section: ## Risks
• Missing section: ## Model Used
• Add the dedup-search checkbox to your PR description and check it once you have searched the GitHub PR list for similar PRs. See the PR template at .github/PULL_REQUEST_TEMPLATE.md and CONTRIBUTING.md → "Before You Start: Search First".
• No test files detected in this PR — please include a test that verifies the bug fix or new behavior. If this PR genuinely doesn't need a test (e.g. a refactor), please retitle with refactor: prefix.

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: dfc58af

Critical Issues (0)

Important Issues (1)

• [native-codex] Dockerfile.agent-toolchain:29 — The image build defaults AMT_PROTOCOL_REF to the mutable main branch, so rebuilding the same toolchain inputs can silently compile and publish a different third-party library (or fail when upstream changes), undermining the images content-addressed/reproducible build contract.
  • Pin this default to a reviewed immutable commit or release tag, and update it deliberately when upgrading the AMT protocol dependency.

Suggestions (0)

Strengths

• Installs both the shared and static AMT libraries plus the header, and verifies the dynamic linker entry and header after installation.
• Moves Cargos writable cache to a node-owned path while leaving the system Rust toolchain shared and read-only.

Recommended Action

1. Fix the Important issue before merge.
2. Re-run the toolchain image build and verify the AMT FFI consumer links successfully.

[allyblockcast]

Hey @eyad-hussein! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• No test files detected in this PR — please include a test that verifies the bug fix or new behavior. If this PR genuinely doesn't need a test (e.g. a refactor), please retitle with refactor: prefix.

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

[eyad-hussein]

Both review items addressed — pushed f0188a1

✅ Important issue: AMT_PROTOCOL_REF pinned to an immutable commit

Agreed, and the reasoning is sharper than my original justification. I had defaulted to main to match multicast's CI, but CI's cache key and an image's content-addressed tag are not the same contract: here the tag is derived from this Dockerfile's bytes, so a floating ref lets identical inputs produce a different library. That defeats the point of the content addressing.

Pinned to ef1bf21a7b82e964dd336b8fe1f7204860c839d5. That is not an arbitrary "current HEAD" — it is exactly the commit multicast's own amt-protocol submodule points at:

$ git ls-tree HEAD amt-protocol          # in Blockcast/multicast
160000 commit ef1bf21a7b82e964dd336b8fe1f7204860c839d5    amt-protocol

So the baked library matches what the consuming repo expects, rather than becoming a third independently-drifting pin. The two must be bumped together, and the Dockerfile comment says so.

Also replaced the branch clone with an explicit shallow fetch of that SHA plus a fail-closed guard:

if [ "${AMT_PROTOCOL_REF}" != "${resolved}" ]; then
  echo "ERROR: AMT_PROTOCOL_REF must be a full immutable commit SHA; ..." >&2
  exit 1
fi

Anything that resolves to something other than itself — a branch, a tag, a short SHA — stops the build. Reproducibility is now enforced rather than conventional, so this cannot silently regress via an --build-arg. Verified git fetch --depth 1 origin <sha> resolves against GitHub before relying on it.

✅ Recommended action 2: re-run the build and verify the consumer links

The image build is the verification and it runs on this PR. Ahead of it, I ran the exact stage inside a pod on the current agent image (paperclip-agent:sha-a1d2f61-k8s-vendored) rather than reasoning from the Dockerfile:

BUILD_OK
    libamt_protocol.so (libc6,x86-64) => /usr/local/lib/libamt_protocol.so
-rw-r--r-- 1 root root     8883  /usr/local/include/amt_protocol.h
-rw-r--r-- 1 root root 22331558  /usr/local/lib/libamt_protocol.a
-rw-r--r-- 1 root root   357784  /usr/local/lib/libamt_protocol.so

⚠️ One honest gap: that proves the library builds, installs and registers with the dynamic linker. It does not yet prove a cgo consumer links against it end to end, because the real consumer — multicast's cmd/caddy/{sender,receiver} — also needs the fec-raptorq submodule, which requires SSH and is out of scope here. The end-to-end proof is BLO-30359's AC 1 (CGO_ENABLED=1 go test ./cmd/caddy/sender ./cmd/caddy/receiver ./flutelib completing on assertions), which becomes runnable on the first agent run after this merges and the fleet auto-bumps. I would rather state that boundary than imply coverage I have not produced.

📝 commitperclip: PR body updated

All required sections added — Thinking Path, Linked Issues (with the bug described in issue-template shape), What Changed, Verification, Risks, Model Used, and the full checklist including the dedup-search box. The dedup search was actually run: agent-toolchain, amt-protocol, amt_protocol, libamt, CARGO_HOME across all states in this repo — no duplicate or in-flight PR touches this file for this purpose.

On the "no test files" check — flagging rather than working around it. There is no unit-test seam for a Dockerfile. The executable assertions here are the build-failing self-checks in the new stage (ldconfig -p | grep -q amt_protocol, test -f …/amt_protocol.h) plus the new AMT_PROTOCOL_REF guard — the same discipline this file already applies with tinygo version, java -version, protoc --version and the ffmpeg -version loop. I have not retitled to refactor: as the bot suggests, because this is a fix and relabelling it to clear a check would be the wrong trade. Maintainer's call if you disagree.

For context on why this matters

Six rows are blocked on the environment, not the code — including two critical ones, and BLO-30253, which gates an unplayable production live delivery service. Every run on them for four days has ended with a disposition naming the toolchain. This is the whole remaining ask on BLO-30359's toolchain half.

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: f0188a1

Prior Findings Dispositioned (1)

• prior:dfc58af important 1 — fixed — Dockerfile.agent-toolchain:122 — AMT_PROTOCOL_REF is now pinned to the full immutable commit SHA ef1bf21a7b82e964dd336b8fe1f7204860c839d5, and the build guard at lines 129-134 rejects any ref that resolves differently.

Critical Issues (0)

Important Issues (0)

Suggestions (0)

Strengths

• Builds and installs the AMT FFI artifacts after the system Rust toolchain is available, with linker and header self-checks.
• Provides a node-owned Cargo cache without changing the shared system Rust installation.
• Keeps the dependency pin and content-derived image tagging aligned with reproducible builds.

Recommended Action

1. No Critical or Important issues found; this review is approved.
2. Consider Suggestions opportunistically.

[allyblockcast]

Hey @eyad-hussein! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• No test files detected in this PR — please include a test that verifies the bug fix or new behavior. If this PR genuinely doesn't need a test (e.g. a refactor), please retitle with refactor: prefix.

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

[allyblockcast]

Hey @eyad-hussein! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• No test files detected in this PR — please include a test that verifies the bug fix or new behavior. If this PR genuinely doesn't need a test (e.g. a refactor), please retitle with refactor: prefix.

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip