Skip to content

fix(security): tolerate Rekor entry conflicts on cosign attest - #373

Open
Cre-eD wants to merge 1 commit into
mainfrom
fix/rekor-409-idempotent
Open

fix(security): tolerate Rekor entry conflicts on cosign attest#373
Cre-eD wants to merge 1 commit into
mainfrom
fix/rekor-409-idempotent

Conversation

@Cre-eD

@Cre-eD Cre-eD commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Problem

A Rekor createLogEntryConflict (HTTP 409) fails the whole deploy:

Error: failed to attach SBOM: cosign attest failed: exit status 1 (stderr:
  signing <image>@sha256:...: signing bundle: error signing bundle:
  [POST /api/v1/log/entries][409] createLogEntryConflict
  {"code":409,"message":"an equivalent entry already exists in the transparency
   log with UUID 108e9186e8c5677a..."}
)
error: update failed

The damaging part is where this lands. sbom-att-* and prov-att-* run after
the workload resources, so Pulumi reports update failed with the new revision
already rolled out and healthy:

Time Event
T+0:00 Deployment updated, revision N→N+1, updatedReplicas: 1
T+1:30 prov-att-* → Rekor 409 ❌
T+1:52 sbom-att-* → Rekor 409 → error: update failed

Pulumi still summarises ~ 8 updated, 9 changes, 25 unchanged. The rollout
succeeded; only the attestation step died. Nothing in the run status conveys
that, so a red deploy sends operators hunting for a rollout problem that isn't
there. Re-running the workflow succeeds, which is the tell that the failure is
transient rather than a config error.

Cause

cosign retries its Rekor upload on a client timeout or 5xx. When the first
attempt already committed server-side, the retry replays a byte-identical body
and Rekor answers 409 — its dedup response. Several deploy jobs attesting
against the public-good instance in the same window makes that likely: in the
run that prompted this, nine jobs started inside 84 seconds and seven failed
this way, all in one burst, with no comparable failure in the preceding 100 runs.

cosign sign already handled this — runCosignSign retried on
isRekorConflict. cosign attest did not, in either the SBOM or the provenance
attacher.

Change

Lift the detector and the retry loop out of keyless.go into
signing.RetryOnRekorConflict and use it from all three cosign call sites
(sign, SBOM attest, provenance attest). isRekorConflict /
maxSignAttempts become IsRekorConflict / MaxCosignAttempts so the sbom and
provenance packages can reach them; both already import signing, so no new
dependency edge.

Retry, not treat-as-success. A 409 means the entry is in the tlog, which is
tempting to call done, but cosign uploads to Rekor before it pushes to the
registry — a tlog entry does not prove the attestation was attached. Swallowing
the 409 would report success on an image with no attestation, which is worse than
a red build. A fresh keyless invocation mints a new ephemeral certificate, so the
replayed body differs and the conflict clears. Deterministic keys reproduce the
same signature and exhaust the loop, which stays a failure — the same reasoning
the existing runCosignSign comment already documents.

The attest commands are rebuilt inside the retry closure, since an exec.Cmd
cannot be run twice.

Tests

  • signing: RetryOnRekorConflict — retry clears a conflict, an unrelated 409
    (registry) fails fast, a persistent conflict exhausts MaxCosignAttempts and
    surfaces the error.
  • sbom / provenance: end-to-end through the real Attach, driving a cosign
    stub on PATH that emits the actual Rekor 409 stderr for the first N
    invocations and counts attempts across process executions.

Verified the new tests fail against the pre-fix code (cosign attest failed on
the first conflict) and pass after. Existing keyless/keybased retry tests updated
for the renames and still pass; full ./pkg/security/... suite green; gofmt
clean.

Follow-ups, not in this PR

Two adjacent issues surfaced while tracing this. Both want their own change:

  1. Shared local artifact paths. resolveSBOMOutputPath returns
    security.sbom.output.local verbatim, so every image in a stack writes one
    file (.sc/artifacts/sbom.json). With concurrent images, sbom-att for image
    A can attach image B's SBOM. resolveProvenanceOutputPath and
    resolveScanOutputPath have the same shape; appendPathSuffix is already the
    established idiom for disambiguating a configured base path. Fixing it changes
    user-visible artifact filenames, so it deserves separate review.
  2. Redundant signing across services sharing an image. A stack whose runs
    list several services backed by one built image signs and attests the same
    digest once per service — doubling Sigstore load, which is what makes conflicts
    likelier. Deduplicating in the Pulumi graph is not straightforward: the digest
    is an unresolved sdk.StringOutput at construction time, so it can't key a
    dedup map there.

A Rekor createLogEntryConflict (HTTP 409) aborted `sc sbom attach` and
`sc provenance attach`, failing the whole Pulumi update. Because both run
after the workload resources, deploys went red with the new revision
already rolled out and healthy — the attestation step was the only
casualty, but operators had no way to tell that from the run status.

cosign retries its Rekor upload on a timeout or 5xx. When the first
attempt already committed server-side, the retry replays a byte-identical
body and Rekor answers 409. Concurrent deploy jobs attesting against the
public-good instance make that likely.

`cosign sign` already handled this via runCosignSign; attest did not.
Lift the conflict detector and the retry loop into RetryOnRekorConflict
and use it from all three call sites. Retry rather than treating 409 as
success: cosign uploads to Rekor before it pushes to the registry, so a
tlog entry does not prove the attestation was attached. A fresh keyless
invocation mints a new ephemeral certificate, so the replayed body
differs and the conflict clears; deterministic keys reproduce the same
signature and correctly exhaust the loop.

Signed-off-by: Dmitrii Creed <creeed22@gmail.com>
@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown

Semgrep Scan Results

Repository: api | Commit: 723ac7c

Check Status Details
⚠️ Semgrep Warning 1 warning(s), 5 total

Scanned at 2026-08-08 09:35 UTC

@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown

Security Scan Results

Repository: api | Commit: 723ac7c

Check Status Details
✅ Secret Scan Pass No secrets detected
⚠️ Dependencies (Trivy) High 1 high, 3 total
⚠️ Dependencies (Grype) High 1 high, 3 total
📦 SBOM Generated 523 components (CycloneDX)

Scanned at 2026-08-08 09:35 UTC

@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown

📊 Statement coverage

Measured on the documented included set (see docs/TESTING.md → Coverage scope). Observe-only — no regression gate is enforced yet.

Scope This PR main baseline Δ
Included set (Gold-tier denominator) 90.4% 90.4% +0.0 pp
Full set (whole repo, transparency) 28.3% 28.3% +0.0 pp

Baseline: main @ 442f351

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant