fix(security): tolerate Rekor entry conflicts on cosign attest - #373
Open
Cre-eD wants to merge 1 commit into
Open
fix(security): tolerate Rekor entry conflicts on cosign attest#373Cre-eD wants to merge 1 commit into
Cre-eD wants to merge 1 commit into
Conversation
A Rekor createLogEntryConflict (HTTP 409) aborted `sc sbom attach` and `sc provenance attach`, failing the whole Pulumi update. Because both run after the workload resources, deploys went red with the new revision already rolled out and healthy — the attestation step was the only casualty, but operators had no way to tell that from the run status. cosign retries its Rekor upload on a timeout or 5xx. When the first attempt already committed server-side, the retry replays a byte-identical body and Rekor answers 409. Concurrent deploy jobs attesting against the public-good instance make that likely. `cosign sign` already handled this via runCosignSign; attest did not. Lift the conflict detector and the retry loop into RetryOnRekorConflict and use it from all three call sites. Retry rather than treating 409 as success: cosign uploads to Rekor before it pushes to the registry, so a tlog entry does not prove the attestation was attached. A fresh keyless invocation mints a new ephemeral certificate, so the replayed body differs and the conflict clears; deterministic keys reproduce the same signature and correctly exhaust the loop. Signed-off-by: Dmitrii Creed <creeed22@gmail.com>
Cre-eD
requested review from
Laboratory,
smecsia and
universe-ops
as code owners
August 8, 2026 09:34
Semgrep Scan ResultsRepository:
Scanned at 2026-08-08 09:35 UTC |
Security Scan ResultsRepository:
Scanned at 2026-08-08 09:35 UTC |
📊 Statement coverageMeasured on the documented included set (see
Baseline: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A Rekor
createLogEntryConflict(HTTP 409) fails the whole deploy:The damaging part is where this lands.
sbom-att-*andprov-att-*run afterthe workload resources, so Pulumi reports
update failedwith the new revisionalready rolled out and healthy:
Deploymentupdated, revision N→N+1,updatedReplicas: 1✅prov-att-*→ Rekor 409 ❌sbom-att-*→ Rekor 409 →error: update failedPulumi still summarises
~ 8 updated, 9 changes, 25 unchanged. The rolloutsucceeded; only the attestation step died. Nothing in the run status conveys
that, so a red deploy sends operators hunting for a rollout problem that isn't
there. Re-running the workflow succeeds, which is the tell that the failure is
transient rather than a config error.
Cause
cosign retries its Rekor upload on a client timeout or 5xx. When the first
attempt already committed server-side, the retry replays a byte-identical body
and Rekor answers 409 — its dedup response. Several deploy jobs attesting
against the public-good instance in the same window makes that likely: in the
run that prompted this, nine jobs started inside 84 seconds and seven failed
this way, all in one burst, with no comparable failure in the preceding 100 runs.
cosign signalready handled this —runCosignSignretried onisRekorConflict.cosign attestdid not, in either the SBOM or the provenanceattacher.
Change
Lift the detector and the retry loop out of
keyless.gointosigning.RetryOnRekorConflictand use it from all three cosign call sites(
sign, SBOMattest, provenanceattest).isRekorConflict/maxSignAttemptsbecomeIsRekorConflict/MaxCosignAttemptsso the sbom andprovenance packages can reach them; both already import
signing, so no newdependency edge.
Retry, not treat-as-success. A 409 means the entry is in the tlog, which is
tempting to call done, but cosign uploads to Rekor before it pushes to the
registry — a tlog entry does not prove the attestation was attached. Swallowing
the 409 would report success on an image with no attestation, which is worse than
a red build. A fresh keyless invocation mints a new ephemeral certificate, so the
replayed body differs and the conflict clears. Deterministic keys reproduce the
same signature and exhaust the loop, which stays a failure — the same reasoning
the existing
runCosignSigncomment already documents.The attest commands are rebuilt inside the retry closure, since an
exec.Cmdcannot be run twice.
Tests
signing:RetryOnRekorConflict— retry clears a conflict, an unrelated 409(registry) fails fast, a persistent conflict exhausts
MaxCosignAttemptsandsurfaces the error.
sbom/provenance: end-to-end through the realAttach, driving acosignstub on
PATHthat emits the actual Rekor 409 stderr for the first Ninvocations and counts attempts across process executions.
Verified the new tests fail against the pre-fix code (
cosign attest failedonthe first conflict) and pass after. Existing keyless/keybased retry tests updated
for the renames and still pass; full
./pkg/security/...suite green;gofmtclean.
Follow-ups, not in this PR
Two adjacent issues surfaced while tracing this. Both want their own change:
resolveSBOMOutputPathreturnssecurity.sbom.output.localverbatim, so every image in a stack writes onefile (
.sc/artifacts/sbom.json). With concurrent images,sbom-attfor imageA can attach image B's SBOM.
resolveProvenanceOutputPathandresolveScanOutputPathhave the same shape;appendPathSuffixis already theestablished idiom for disambiguating a configured base path. Fixing it changes
user-visible artifact filenames, so it deserves separate review.
runslist several services backed by one built image signs and attests the same
digest once per service — doubling Sigstore load, which is what makes conflicts
likelier. Deduplicating in the Pulumi graph is not straightforward: the digest
is an unresolved
sdk.StringOutputat construction time, so it can't key adedup map there.