fix(ra): publish certificate renewal snapshots atomically

[csnitker-godaddy]

Successful identity rotation and server renewal publish a signed AGENT_RENEWED event in the corresponding V1/V2 format. Each event carries the complete certificate overlap set. Retries preserve the signed event, and events for the same agent are delivered in order.

A replacement BYOC certificate becomes live only after domain proof succeeds. Failed or cancelled submissions cannot enter the live certificate set. Failed completion or cancellation leaves the previous consistent state intact; cancellation also preserves a concurrent completion or replacement.

Renewing one certificate family succeeds when the other has expired. The event retains the lapsed family's last-expiring non-revoked certificate with its original fingerprint and expiry, so the TL continues to report EXPIRED until both required families are usable. An old overlapping certificate does not shorten the validity of its replacement.

Contract comparison

The existing ProducerEvent in spec/api-spec-tl-v2.yaml already defines:

eventType:
  type: string
  enum: [AGENT_REGISTERED, AGENT_RENEWED, AGENT_DEPRECATED, AGENT_REVOKED]

The served V1 and V2 event schemas define:

"renewalStatus": {
  "description": "Status of certificate renewal",
  "enum": ["FAILED", "PENDING", "SUCCESS"],
  "type": "string"
}

The existing RenewalVerificationResponse in spec/api-spec-v2.yaml remains:

type: object
properties:
  status:
    type: string
    enum: [VERIFIED, ISSUING_CERTIFICATE, COMPLETED]
  csrId:
    type: string
    format: uuid
    nullable: true
  tlsaDnsRecord:
    $ref: '#/components/schemas/DnsRecord'
    nullable: true
  nextStep:
    $ref: '#/components/schemas/NextStep'
required:
  - status
  - nextStep

No handler response fields, routes, event fields, or signing inputs change. The event uses renewalStatus: SUCCESS; the HTTP response retains status: COMPLETED. Identity CSR submission retains the existing 202 response and csrId. V1 keeps the singleton and valid*Certs fields, and V2 keeps its unified certificate arrays.

Expiry and lapsed-family semantics depend on agentnameservice/ans-registry#68. Registry ANS-1 prose describing AGENT_RENEWED as reserved/unemitted still needs clarification.

Renewal events attest DNS observed before renewal completes. This does not implement later TLSA/DNS re-verification or resealing; that remains agentnameservice/ans-registry#66.

Validation

make build, make check, and make test-race pass on this branch. Coverage is internal 90.73%, domain 100.00%, crypto 98.45%. The CI dependency-license check passes.

Regressions cover V1/V2 CSR, BYOC, and identity renewal, certificate overlap, signed retries, failure rollback, event ordering, and cancellation races. Eight RA-to-TL lapse cases verify successful partial renewal, retained expiry evidence, an EXPIRED badge with no status token, and recovery after the other family renews.

Local HTTP acceptance passes on both API versions. It covers registration, CSR renewal, identity rotation, rejection and cancellation of unproven BYOC input, verified BYOC completion, and duplicate replay. Offline verification checks receipts, checkpoints, and status tokens containing two identity and three server certificates per agent.

This is local acceptance coverage. Live Let's Encrypt, public DNS, and the selected OIDC provider still require the deployment acceptance run.

Fourth PR in the deployment-fix stack, dependent on the ACME and TL changes.

Fixes #124.

Depends on #127; this PR targets fix/tl-ingestion-status to keep its review diff scoped.

AI assistance

Assisted-by: Codex (GPT-6), under Connor Snitker's direction.

[kperry-godaddy]

The transactional core is right: certificate, CSR/renewal state and the signed AGENT_RENEWED row commit in one uow.Run with txCtx threaded to every store call including the outbox, signed bytes are built once and replayed verbatim, BYOC PEM stays on the pending renewal until proof, and the in-tx re-reads close the cancel/complete race. The one thing I would hold the merge for is the new per-agent gate in Claim: it sits on a worker that never dead-letters, so a single permanently rejected AGENT_RENEWED row now holds that agent's AGENT_REVOKED forever.

Three things I'd fix before this ships, all inline (two with suggestions you can commit as-is). Non-blocking notes below.

Not blocking, worth a look

• commitCertificateRenewal observes DNS after FinalizeOrder has already issued; a verifier failure then returns 500 with nothing persisted, orphans the issued certificate and forces a repeat finalize (the self-CA mints a new serial each time). Observing before the CA call in finalizeCSRRenewal and finalizeBYOCRenewal, and passing the evidence in, shrinks the post-issuance failure window to the local transaction.
• Identity rotation now hard-fails on a DNS-verifier error although ANS-1 §7 says rotation needs no fresh domain proof; observation is evidence, not a gate. Worth stating the policy either way and pinning it with a test (erroringDNSVerifier already exists).
• Identity rotation reuses AGENT_RENEWED while internal/tl/event/event.go:79-81 still documents the type as server-renewal-only and ans-registry#65 lists the decision as open; once sealed, those leaves are immutable. The doc comments can change in this PR; the registry text should follow before many leaves carry it.
• The V1 route rebinding (SubmitIdentityCSRV1, VerifyRenewalACMEV1) has no handler-level test, so reverting either call keeps the suite green while V1 clients' renewals land on the V2 lane; v1lifecycle_test.go already shows the outbox-lane assertion. The *V1 twins also introduce a second lane-selection convention beside VerifyInput{SchemaVersion}; one convention would be easier to live with.

[kperry-godaddy]

The NOT EXISTS gate looks only at sent_at_ms, not at whether the earlier row can ever succeed, and the worker never dead-letters: every non-429 4xx is PermanentError and MarkFailed only reschedules at max backoff, so sent_at_ms stays NULL forever. One permanently rejected AGENT_RENEWED (producer-key rotation, a RaID change, STALE_AGENT_EVENT after a clock step, schema drift between RA and TL versions) now holds every later row for that agent, including AGENT_REVOKED. The RA reports REVOKED locally while the TL, offline verifiers and the Finder keep the agent live until someone edits SQLite by hand, and nothing logs the held rows. With an unsent RENEWED head at attempts=40 and a later REVOKED for the same agent, this query never returns the REVOKED row; the previous query does. Before this change a poisoned row only cost itself.

I'd hold the merge for this one. Two parts. First, terminal events should never wait behind a renewal; the TL rejects anything after AGENT_REVOKED, so REVOKED-first is the safe order:

Suggested change

	FROM outbox_events AS candidate
	WHERE sent_at_ms IS NULL AND next_attempt_at_ms <= ?
	AND NOT EXISTS (
	SELECT 1 FROM outbox_events AS earlier
	WHERE earlier.agent_id = candidate.agent_id
	AND earlier.id < candidate.id
	AND earlier.sent_at_ms IS NULL
	)
	ORDER BY id ASC
	LIMIT ?`
	FROM outbox_events AS candidate
	WHERE sent_at_ms IS NULL AND next_attempt_at_ms <= ?
	AND (candidate.event_type = 'AGENT_REVOKED' OR NOT EXISTS (
	SELECT 1 FROM outbox_events AS earlier
	WHERE earlier.agent_id = candidate.agent_id
	AND earlier.id < candidate.id
	AND earlier.sent_at_ms IS NULL
	))
	ORDER BY id ASC
	LIMIT ?`

Second, a terminal row state the predicate excludes: migration 014 adds dead_at_ms INTEGER, the 013 partial index is rebuilt with AND dead_at_ms IS NULL, both the outer WHERE and the subquery skip dead rows, and a MarkDead(ctx, id, reason) is called by the worker for malformed rows and after a bounded number of permanent attempts, logging at ERROR with id, agentId and eventType. A per-tick summary (pending, held back, oldest age) makes the block visible. A service-level test that completes a renewal, marks its row failed with a permanent reason, calls Revoke and asserts Claim returns the AGENT_REVOKED row pins the behaviour.

[kperry-godaddy]

This keeps only Found records. The lookup verifier folds SERVFAIL, REFUSED and timeouts into per-record Error with Found=false and a nil call error, so a degraded resolver lets identity rotation and renewal completion succeed while the sealed AGENT_RENEWED carries a narrowed or empty dnsRecordsProvisioned[], with no log line. It also drops the DNSSEC-authenticated mismatch evidence that verifyDNSRecords treats as tampering on the registration path. The leaf is signed and append-only, so the first signal is a verifier or the Finder reading a leaf that misdescribes the zone, days later. attestedDNSRecords and droppedForLookupError already implement the classification and the WARN discipline for the same result set; reusing them keeps the two paths consistent:

Suggested change

	evidence.results = result.Results
	for _, r := range result.Results {
	if r.Found {
	evidence.records = append(evidence.records, r.Record)
	}
	}
	// The verifier ran, so nil Results means nothing was observed;
	// attestedDNSRecords reads nil as "not consulted" and would attest
	// every expected record.
	evidence.results = result.Results
	if evidence.results == nil {
	evidence.results = []port.RecordVerification{}
	}
	attested, dropped := attestedDNSRecords(expected, evidence.results)
	evidence.records = attested
	if len(dropped) > 0 {
	ev := s.logger.Info()
	if droppedForLookupError(dropped) {
	ev = s.logger.Warn()
	}
	ev.Str("agentId", reg.AgentID).
	Str("fqdn", reg.FQDN()).
	Int("expectedCount", len(expected)).
	Int("attestedCount", len(attested)).
	Interface("droppedRecords", dropped).
	Msg("omitting unverified DNS records from the renewal attestation")
	}

Worth fixing before this ships. Whether a DNSSEC-authenticated mismatch on TLSA/SVCB/HTTPS should fail the renewal (as registration does) or log at ERROR is a policy call; either beats silence. A test driving Found=false with Error set through rotation and renewal would pin it.

[kperry-godaddy]

certificate_events.go imports no zerolog and never touches s.logger, and the same holds for commitCertificateRenewal, submitIdentityCSR, CancelServerCertRenewal and finalizeBYOCRenewal: the RA's sole authority for binding a certificate fingerprint to an ANS name now commits certificates, CSR state and outbox rows with no INFO line, and returns bare errors on the in-tx AGENT_NOT_ACTIVE / RENEWAL_NOT_PENDING rejections, verifier failures and store faults. The handler logs non-domain 500s without agentId, fqdn or renewal id, and nothing below 500. RegistrationService.logger is wired and already used by the activation seal.

Worth fixing before this ships; CLAUDE.md treats a silent component as a defect. For this function: ERROR with .Err(err), agentId, fqdn and schemaVersion on the publication-unavailable and enqueue-failure paths, and after a successful enqueue

	s.logger.Info().
		Str("agentId", reg.AgentID).
		Str("fqdn", reg.FQDN()).
		Str("schemaVersion", schemaVersion).
		Str("eventType", string(event.TypeAgentRenewed)).
		Int("dnsRecordsAttested", len(evidence.records)).
		Msg("certificate renewal event enqueued")

In commitCertificateRenewal (renewal.go:490-534) wrap the uow.Run result the same way (ERROR for store and signer faults, WARN for the in-tx domain rejections, both with renewalId), and on success log renewalId, renewalType, fingerprint, notAfter and schemaVersion; mirror it in submitIdentityCSR and CancelServerCertRenewal.

[Copilot]

Copilot review overview

🟡 Changes recommended

A permanently undeliverable outbox event can block all later events for an agent; the store failure paths also need structured logging.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1

Open (1)

• Permanent outbox failures can block all newer agent snapshots · New

What changed in this PR

This PR atomically publishes certificate-renewal snapshots and signed V1/V2 AGENT_RENEWED events while preserving rollback, overlap, expiry, and ordering behavior.

Changes:

• Adds transactional renewal and cancellation handling.
• Adds BYOC proof gating and certificate-lapse semantics.
• Enforces per-agent outbox ordering with regression coverage.
• Adds deployment and wire-format documentation.

File	Summary
internal/​ra/​service/​v1event.go	Builds complete V1 certificate snapshots.
internal/​ra/​service/​renewal.go	Implements transactional renewal and cancellation.
internal/​ra/​service/​renewal_cancel_test.go	Tests cancellation race behavior.
internal/​ra/​service/​registration_test.go	Covers registration-related renewal behavior.
internal/​ra/​service/​order_flow_test.go	Tests renewal ordering flows.
internal/​ra/​service/​lifecycle.go	Publishes identity rotation events atomically.
internal/​ra/​service/​helpers.go	Computes certificate expiry semantics.
internal/​ra/​service/​helpers_test.go	Tests certificate helper behavior.
internal/​ra/​service/​deployment_regression_test.go	Covers deployment regressions.
internal/​ra/​service/​certificate_snapshot_test.go	Tests certificate snapshots and overlap.
internal/​ra/​service/​certificate_lapse_test.go	Tests lapsed-family handling.
internal/​ra/​service/​certificate_events.go	Constructs and publishes renewal events; store failures need structured logging.
internal/​ra/​service/​certificate_events_test.go	Tests certificate event publication.
internal/​ra/​handler/​v1renewal.go	Routes V1 renewal flows.
internal/​ra/​handler/​v1renewal_test.go	Tests V1 renewal handling.
internal/​ra/​handler/​v1lifecycle_test.go	Tests V1 lifecycle behavior.
internal/​ra/​handler/​v1certificates.go	Routes V1 certificate operations.
internal/​ra/​handler/​v1certificates_test.go	Tests V1 certificate handling.
internal/​ra/​handler/​lifecycle.go	Updates lifecycle handling and documentation.
internal/​adapter/​store/​sqlite/​outbox.go	Enforces per-agent outbox ordering; terminal failures can block later events indefinitely.
internal/​adapter/​store/​sqlite/​outbox_order_test.go	Tests outbox retry ordering.
internal/​adapter/​store/​sqlite/​migrations/​013_outbox_agent_order.sql	Adds the outbox ordering index.
docs/​operations/​review-wire-comparison.md	Documents wire-format compatibility.
docs/​operations/​deployment-fix-upgrade.md	Documents deployment and recovery guidance.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.