Skip to content

Feat/sdk canonical offer bytes helper - #28

Merged
legendko merged 21 commits into
mainfrom
feat/sdk-canonical-offer-bytes-helper
Jul 24, 2026
Merged

Feat/sdk canonical offer bytes helper#28
legendko merged 21 commits into
mainfrom
feat/sdk-canonical-offer-bytes-helper

Conversation

@legendko

@legendko legendko commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

Export helpers.CanonicalOfferBytes

Summary

Export the offer canonicalization the Go SDK already uses internally, so a downstream Exchange can persist the exact bytes an Offer signature covers and re-verify them verbatim — instead of re-implementing (and drifting from) the canonical form.

Why

The bytes an Offer signature is computed over come from one recipe — clone the offer, clear signature/signature_algorithm, render canonical proto-JSON, apply RFC 8785 JCS — shared by the signer and the verifier. No exported Go function returned those bytes. Persisting re-verifiable offer evidence would otherwise mean duplicating a security-critical canonical form downstream; any later drift (a new Offer field, a tweaked option, a JCS bump) would silently break re-verification of already-stored evidence. Exposing the existing function keeps exactly one definition.

What changed

  • New export helpers.CanonicalOfferBytes(offer *rampv1.Offer) ([]byte, error) — the previously-unexported canonicalOfferPayload, body unchanged (nil-check → proto.Clone → clear the two signature fields → JCS-over-proto-JSON). expires_at stays covered; only signature/signature_algorithm are cleared.
  • One code pathSignOffer, VerifyOffer, and the wire-canonical vector generator all route through it, so signer and verifier provably share one canonicalization.
  • Cross-language parity — registered in sdk/parity/symbol-map.json as a mapped symbol against the equivalents Python (canonical_offer_payload) and TypeScript (canonicalOfferPayload) already export publicly; docs/sdk-parity-matrix.md regenerated (73 parity / 98 exclusions).
  • Docs — changelog entry in proto/CHANGELOG.md and its website mirror; godoc notes that re-verification also needs the stored Offer.signature and the signer's public key (the canonical bytes are the signed message, not sufficient alone).
  • Tests — byte-parity with SignOffer, signature-field independence, expires_at coverage, nil handling, and non-mutation of the caller's offer.

Scope

Additive and non-breaking: no .proto, no gen/, no wire change, no new dependency (gowebpki/jcs already present). No CanonicalAcceptanceBytes — acceptance re-verifies through the already-exported VerifyOfferAcceptance.

Testing

  • go build ./... && go vet ./sdk/go/... && go test ./sdk/go/... — green; golden offer/acceptance vector replays byte-identical, so behavior is preserved.
  • ./scripts/ci-local.sh — pass (parity-matrix drift, Pydantic/Zod type parity, conformance).
  • API-surface parity gate (test_api_surface_parity.py + test_parity_matrix_generated.py) — 11 passed.
  • website build — pass.

Downstream

Additive; consumers pick it up by bumping the github.com/RAMP-Protocol/protocol commit pin. No cross-repo alignment needed — the exported bytes match what the downstream already signs and verifies (expires_at included).


Update (2026-07-23) — the branch now exports both canonical-bytes accessors

Everything above describes the branch at its first two commits and is kept for history. Three statements in it are now false; they are corrected below. Suggested retitle: Export helpers.CanonicalOfferBytes + helpers.CanonicalAcceptanceBytes.

Corrections to the text above

Above Now
Title — "Export helpers.CanonicalOfferBytes" The branch exports both canonical-bytes accessors, offer and acceptance
"docs/sdk-parity-matrix.md regenerated (73 parity / 98 exclusions)" 74 symbols at cross-language parity · 14 documented divergences · 98 Go-idiomatic exclusions · 23 conformance corpora
"No CanonicalAcceptanceBytes — acceptance re-verifies through the already-exported VerifyOfferAcceptance." Reversed. That argument sized the risk as the field set (four fixed strings, drift-resistant). The real risk is the canonicalization recipe, and the recipe has already changed twice
Scope — "no .proto, no gen/" Still no .proto message, field, or wire change, but proto comments did change, which regenerates gen/

Why CanonicalAcceptanceBytes after all

Re-deriving the bytes at verification time pins an already-signed payload to whatever canonicalization the SDK implements later. The failure mode is silent and indistinguishable from fraud: "the agent's signature does not verify" reads exactly like "the agent never signed".

That is not hypothetical here. The acceptance canonical form has changed twice:

  1. deterministic protobuf binary → RFC 8785 JCS over proto-JSON;
  2. camelCase json_name keys → snake_case proto names, which re-signed the golden vectors — signatures produced over the camelCase-JCS form no longer verify.

A party keeping evidence must store the exact signed bytes and re-verify against them verbatim. Both accessors now make that possible; docs/design-history.md records the reasoning.

What changed since the original description

  • New export helpers.CanonicalAcceptanceBytes(offer *rampv1.Offer, requester *rampv1.Requester, idempotencyKey string) ([]byte, error) — the previously-unexported canonicalAcceptancePayload, body byte-identical (the non-comment diff is the func line plus three call sites). RFC 8785 JCS over canonical proto-JSON of AgentAcceptancePayload{offer_sig, requester_id, requester_domain, idempotency_key}.
  • One code pathSignOfferAcceptance, VerifyOfferAcceptance, and the golden-vector emitter all route through it. No unexported duplicate remains.
  • Cross-language parity — registered in sdk/parity/symbol-map.json as a mapped symbol with both faces filled (python: jcs_acceptance_payload, ts: acceptancePayload, allowlist_reason: null) — deliberately not a go_exclusions entry. Python and TS already export the equivalent public accessor, so filing it as Go-only would have been the same mistake this branch's earlier commit had to reverse for CanonicalOfferBytes. BASELINE_ALLOWLIST stays 14.
  • Normative spec text correctedAgentAcceptance and AgentAcceptancePayload still described the retired deterministic-protobuf form while the canonical-signing block on Offer.signature in the same file already stated that JCS over proto-JSON "applies to the agent offer-acceptance signature". An implementation following the acceptance text would have produced non-verifying signatures. The acceptance text now points at that single definition: the message fixes the FIELD SET, Offer.signature fixes the BYTE LAYOUT. Comments only — no field, message, or wire change; gen/ regenerated with the pinned buf 1.66.1 and the hand-maintained website mirror updated.
  • Review follow-ups — the retired-form sweep finished where it had been left short: two more sites inside CI-run Go code, plus three comments still describing the JCS rendering as camelCase (one of them describing the canonical-signing option set, which is UseProtoNames: true). The vacuous TestAgentAcceptancePayload_deterministicMarshal is replaced by a field-set guard: canonical proto-JSON omits unpopulated fields, so a field added to AgentAcceptancePayload without a matching assignment in CanonicalAcceptanceBytes would drop out of the signed bytes with the vectors byte-identical and every gate green.
  • Docs — changelog entries in proto/CHANGELOG.md and its website mirror for the export and the spec-text correction; a docs/design-history.md section on why JCS over proto-JSON replaced deterministic protobuf binary, and why the canonical bytes are consequently a public accessor in all three SDKs.
  • Tests — byte-parity with SignOfferAcceptance; byte-identity against every vector in the committed acceptance-vectors.json corpus (which the TS and Python faces already replay, so Go↔TS↔Python byte-identity is pinned by construction); fail-closed on nil offer, nil requester, and an unsigned offer; non-mutation of the caller's messages via proto.Clone/proto.Equal; and the new field-set guard.

Behaviour is unchanged

sdk/go/helpers/testdata/acceptance-vectors.json is byte-identical to its state at the branch point — RAMP_UPDATE_VECTORS=1 was never run — and its Go, Python, and TS replays all pass. go.mod, go.sum, and canonicalsign.go are untouched across every commit.

Testing (re-run on the final tree)

  • go build ./... && go vet ./... && go test ./sdk/go/... — green.
  • ./scripts/ci-local.shCI-local: PASS (gen drift, corpus, parity-matrix drift, Pydantic/Zod type parity, conformance).
  • API-surface parity + parity-matrix (CI-only) — 11 passed.
  • sdk/python/tests — 514 passed.
  • sdk/ts && npm test — 47 files / 506 tests.
  • website && npm run build — 79 pages.
  • gen/ reproduces byte-identical from the pinned buf 1.66.1.

Follow-ups filed (not in this PR)

Offer.signature is documented as a JWS across 15 sites (4 in ramp.proto, 7 website pages, docs/design-history.md, CLAUDE.md, a downstream design doc) but is a bare hex-encoded detached Ed25519 signature everywhere it is implemented. No ADR ever decided JWS and no JWS signer has ever existed in either repo; the committed corpus carries a 128-character all-hex signature. Documentation-only, but it makes a conforming implementer produce non-verifying signatures, so it gets its own change rather than riding along here.
Attestation.signature envelope is genuinely undecided (the field's own comment, the file header, and the absence of any implementation disagree). Scoped onto the attestation implementation ticket rather than guessed at in a documentation sweep.
TS and Python omit only requester_domain when empty while Go omits every empty field; Requester.id carries no min_len, so "" is wire-valid and the three SDKs would sign different bytes. Fails closed (mismatch, never bypass) and pre-dates this branch, but it needs a new corpus vector, which this branch cannot regenerate without breaking its own behaviour-unchanged guarantee.

legendko added 18 commits July 22, 2026 18:55
Expose the previously-unexported offer canonicalization as CanonicalOfferBytes: RFC 8785 JCS over canonical proto-JSON with signature/signature_algorithm cleared and expires_at included -- the exact bytes an Offer signature is computed over. Callers persisting a signed offer can now store and re-verify those bytes verbatim, independent of how the Offer message later evolves. SignOffer and VerifyOffer route through it, so there is a single canonicalization path and behavior is unchanged (conformance vectors byte-identical).

Go-only: Python/TS have no consumer for the raw signed bytes and route offer signing/verification through their sign/verify faces, so the symbol is registered in symbol-map.json go_exclusions and the parity matrix regenerated. No proto or wire change.
…y matrix

CanonicalOfferBytes was registered as a Go-only exclusion, but Python
(canonical_offer_payload) and TypeScript (canonicalOfferPayload) already export
the exact equivalent as curated public symbols. Reclassify it as a mapped symbol
and regenerate the matrix (72->73 parity, 99->98 exclusions), so the generated
parity doc no longer asserts a cross-language face is absent when it exists.

Alongside:
- refresh the two "Go oracle" byte-identity comments in the Python/TS cores to
  the exported name (canonicalOfferPayload -> CanonicalOfferBytes)
- record the export in proto/CHANGELOG.md and its website mirror, dropping the
  inaccurate "Go-only" framing and a non-existent sign_offer reference
- note in the godoc that re-verification also needs the stored Offer.signature
  and signer key; the canonical bytes are the signed message, not sufficient alone
- assert CanonicalOfferBytes does not mutate the caller's offer (clears on a clone)
Expose the previously-unexported acceptance canonicalization as
CanonicalAcceptanceBytes: RFC 8785 JCS over canonical proto-JSON of
AgentAcceptancePayload{offer_sig, requester_id, requester_domain,
idempotency_key} -- the exact bytes an agent's offer-acceptance signature is
computed over. A caller persisting an acceptance can now store and re-verify
those bytes verbatim instead of re-deriving them at verification time, which
would pin an already-signed acceptance to whatever canonicalization the SDK
implements later; that form has already changed once, from deterministic
protobuf to JCS over proto-JSON. SignOfferAcceptance, VerifyOfferAcceptance and
the golden-vector emitter route through it, so there is one canonicalization
path and behavior is unchanged (acceptance vectors byte-identical).

Python (jcs_acceptance_payload) and TypeScript (acceptancePayload) already
export the equivalent public accessor, so the symbol is registered in
symbol-map.json as a mapped symbol -- not a Go-only exclusion -- and the parity
matrix regenerated (73->74 parity). No proto or wire change.
…roto-JSON

AgentAcceptance and AgentAcceptancePayload still described the RETIRED signing
form -- "the deterministic protobuf serialization of AgentAcceptancePayload",
"derive BYTE-IDENTICAL signed bytes ... via proto.Marshal(Deterministic: true)"
-- while the canonical-signing block on Offer.signature in the same file already
states that RFC 8785 JCS over canonical proto-JSON "applies to the agent
offer-acceptance signature". The normative spec asserted both, and an
implementation that followed the acceptance text would have produced
non-verifying signatures.

Point the acceptance text at that single definition instead of restating a
superseded recipe: AgentAcceptancePayload fixes the FIELD SET, Offer.signature
fixes the BYTE LAYOUT. Four sites, comments only -- no field, message, or wire
change; gen/ regenerated (buf 1.66.1) and the hand-maintained website mirror
updated, both of which carried the same stale claim.

Also record the reversal in design-history: why deterministic protobuf binary
was rejected as a cross-language signing primitive (protobuf disclaims byte
stability across languages and versions, and it forces every verifier to link a
binary codec), and why the canonical bytes are consequently a public accessor in
all three SDKs -- a form that changed once can change again, so stored evidence
must re-verify against the bytes as signed, not as later re-derived.
… claims

Two comment sites still described the acceptance signed bytes as deterministic
protobuf, and one test executed that form -- inside the same package as
canonicalsign.go, which states the opposite. A third set of comments still
described the JCS rendering as camelCase, retired when the signed form was
re-pinned to snake_case proto names.

- acceptance_payload_test.go: drop TestAgentAcceptancePayload_deterministicMarshal.
  It marshalled the same message twice with Deterministic:true and asserted the
  results matched -- vacuous, and it advertised the superseded recipe. Replace it
  with a field-set guard: AgentAcceptancePayload fixes the FIELD SET the signature
  covers, and canonical proto-JSON omits unpopulated fields, so a field added
  without a matching assignment in CanonicalAcceptanceBytes drops out of the signed
  bytes with the golden vectors byte-identical and every gate green.
- reachability_test.go: the AgentAcceptancePayload out-of-band rationale said the
  signer and verifier "deterministically marshal" it; it is the field set they
  canonicalize under RFC 8785 JCS over proto-JSON.
- gen_vectors_test.go: three comments claimed camelCase. One described the pinned
  canonical-SIGNING option set, which is UseProtoNames:true; one claimed camelCase
  json_names on the wire form three lines above its own option set stating
  snake_case; one credited JCS with preserving a camel shape. Both sides are
  snake_case, so what a from-wire canonicalizer undoes is the zero-inflation and
  the signature fields, not the naming.

Comments and one test swap only -- no behavior change.
…plit

The proto now pins the acceptance signed bytes in two halves -- the message fixes
the FIELD SET, Offer.signature fixes the BYTE LAYOUT -- but the SDK package header
still carried the conflation the proto shed, and design-history recorded only the
first of the two canonical-form reversals.

- acceptance.go: the package header said the signed bytes are pinned by the
  AgentAcceptancePayload message; state both halves. CanonicalAcceptanceBytes'
  godoc said "the transaction's idempotency key" without saying which -- name the
  enclosing execute request, since a TransactionItem carries neither the requester
  nor the key.
- acceptance_test.go: the non-mutation test spot-checked four getters and spent
  half its body asserting determinism. CanonicalAcceptanceBytes clones nothing, so
  compare whole messages with proto.Clone + proto.Equal; determinism is already
  pinned by the corpus test's exact-bytes assertion.
- test_acceptance_jcs_repin.py: the historical re-pin narrative named a Go symbol
  that no longer exists. Keep the history, note the export.
- design-history.md: the canonical form changed TWICE, not once. The second change
  re-pinned the JCS rendering from camelCase json_names to snake_case proto names
  and re-signed the golden vectors -- signatures over the camelCase form no longer
  verify. A document that exists to record reversals was missing one, and it is the
  second data point the "a form that changed can change again" argument rests on.
  Also scope the opening claim: the two payloads that moved off deterministic
  protobuf are the two that cover a proto message; Attestation.signature outlives
  an HTTP exchange too but signs a hand-built JSON object under its own rule.
… pointer

The AgentAcceptance block pointed at the canonical-signing definition as "defined
once on Offer.signature above and applying verbatim here". Both halves were wrong
in effect.

"above" is a position in the .proto, and the comment does not stay there: it is
compiled into gen/ts/wire/schemas.ts, which is ordered alphabetically, so
AgentAcceptanceSchema lands near the top of the file and OfferSchema far below it.

"verbatim" overstates. The Offer.signature formula is JCS(protojson(msg with
signature + signature_algorithm cleared)); AgentAcceptancePayload has neither
field, so the clear step is a no-op there. Say so explicitly instead -- the
definition reduces to JCS(protojson(AgentAcceptancePayload)) -- which is more useful
to an implementer than an instruction to apply a formula whose first step does
nothing.

Comment only -- no field, message, or wire change. gen/ regenerated (buf 1.66.1;
gen/python/wire is unaffected, it carries field descriptions only) and the
hand-maintained website mirror updated with the same reduction note.
The module docstring described the RAMP Connect wire as camelCase protojson and
cited sdk/go/connectserver/codec_test.go as pinning "camelCase, NOT UseProtoNames".
That test's own header pins the opposite, and codec.go marshals with
UseProtoNames=true. The claim was written hours before the codec moved to snake_case
proto names and was never swept; the Go side of the same claim was corrected, the
Python face was not.

Both the wire and the signed form are snake_case, so what the inversion undoes is the
zero-inflation and the signature fields, not the naming. The retired lowerCamel
json_name form stays tolerated on input and is now stated as such -- accepted when it
arrives, never emitted -- which is why _snake stays and is documented as idempotent
on snake keys, and why the offer_camel parameter name still reads correctly.

The live-captured camelCase fixture is reframed as the pre-flip capture it is, kept
as the tolerance case; the synthetic camelCase dicts in the enum-pruning and
set-empty tests are noted as the same. Parity for the current wire is the drift-gated
wire-canonical-vectors corpus, which the test file already replays but the docstring
did not list.

Docstrings only -- no behavior, no identifiers, no fixtures, no generated files.
… payload

Python jcs_acceptance_payload and TS acceptancePayload assemble the
AgentAcceptancePayload JSON object key by key and dropped only an empty
requester_domain, always emitting requester_id and idempotency_key. Go renders
the same object through protojson with EmitUnpopulated=false, which omits EVERY
unpopulated field, so the three SDKs signed different bytes whenever
requester_id was empty — a wire-valid input, since Requester.id carries no
min_len.

The mismatch failed closed (a byte difference, never a bypass), but it
falsified the byte-equivalence the canonical-bytes accessors are registered as
parity symbols to promise. Both hand-built faces now drop each empty string
field.

The corpus gains an empty_requester_id vector so the agreement is pinned rather
than asserted. The change is additive only: the existing vectors and their
signatures are byte-identical, so no already-issued signature is affected.
…er_payload

The wire_canon module and test docstrings claimed the wire-to-canonical
inversion undoes "the zero-inflation and the signature fields". It does not:
from_wire_offer keeps signature/signature_algorithm verbatim (a non-empty value
survives the presence check), and canonical_offer_payload strips them on the way
into JCS — which is why the tests here strip the two keys by hand before
comparing. Misattributing a security-relevant step invites a reader to treat
from_wire_offer output as the pre-JCS signed object.

Also shift the re-pin suite's two present-tense claims about the acceptance
corpus to past tense. Both described the file as still carrying the retired
protobuf-binary form; it has carried canonical_jcs since the re-pin landed, so
the preserved red-step narrative now reads as history rather than current state.
TestCanonicalAcceptanceBytes_matchesCorpus re-declared the committed corpus
schema field for field and asserted an equality the golden emitter already
gates: TestGenerateVectors rebuilds acceptance-vectors.json through
CanonicalAcceptanceBytes and byte-compares the whole committed file on every
default run. Two structs describing one file drift apart on the next
corpus-shape change, which is exactly what the canonicalization re-pin just did.

The exported symbol keeps its external-package coverage through the byte-parity,
fail-closed and non-mutation tests. A comment records why corpus byte-identity
is not asserted a second time here.
…e definition

Both accessors deferred the pinned proto-JSON option set to the unexported
canonicalSignPayload and to canonicalsign.go by filename — neither of which a
godoc reader can resolve. They now name the decisive options inline and point at
the single normative statement of the form, the Offer.signature comment in
ramp.proto, rather than restating the recipe a third time.

The two godocs also stated the persistence guarantee along different axes ("how
the Offer message later evolves" against "how the canonical form later
evolves"). The canonical-form axis is the one the design history argues, so it
is primary in both; the offer keeps its message-evolution clause.

Both now close the evidence contract: a passing signature proves only that the
key holder signed those exact bytes, so a caller must also match the parsed
content against the transaction under dispute — otherwise any valid triple
minted under the same key passes as evidence for any transaction. The design
history says the same, and additionally records why omit-unpopulated obliges a
hand-built payload to enumerate the omission per field.

Also corrects the attestation message name in the design history: the message is
ResourceAttestation; no Attestation message exists.
An Offer from a peer built against a newer schema keeps its unrecognized fields
as protobuf unknown fields, but proto-JSON does not render them — so they never
reach the canonical bytes. That is the safe direction (the peer's signature
covered more than we can reconstruct, so it is rejected rather than verified
over a truncated message), but nothing pinned it, unlike the equivalent limit
the Python canonicalizer documents and tests.

The test builds the newer-peer message by appending an unknown field to the
encoded form, proves it survives unmarshal, proves it does not reach the
canonical bytes, and proves a signature over the peer's own rendering is
rejected. A signature over the reconstructable bytes still verifies, so the
rejection is attributable to the extra signed member.
…l signing

proto-JSON renders only what the schema defines, so a message carrying unknown
fields canonicalizes to bytes that silently drop them. Both directions of that
gap were wrong, and only one of them was ever acknowledged: a signer built
against a newer schema covered more than this build can reconstruct (rejected,
but for a misleading reason), and an on-path party could APPEND unknown fields
to an already-signed Offer without disturbing its signature — unauthenticated
bytes riding on a message the recipient treats as verified, surviving a
proto.Marshal round-trip onward.

canonicalSignPayload now refuses such a message. The walk is written out rather
than delegated to a traversal helper: this is a signature-coverage guard, so its
reachability rules belong where they can be audited. Unknown-field sets are per
message, so it visits nested messages and every element of a repeated or map
field — a top-level check alone passes a payload whose tampering sits one level
down.

VerifyOffer surfaces the refusal as ErrOfferSignatureInvalid rather than
propagating it raw: callers branch on that sentinel to map a rejection to its
denial reason, and a message that arrived carrying extra bytes is a tampered
offer, not an internal fault. SignOffer and CanonicalOfferBytes return it
directly — signing or persisting such a message is a caller bug.

No legitimate traffic changes behaviour. An offer signed WITH a field this build
cannot render already failed to verify, because the renderer dropped it and the
bytes differed; the refusal only makes the reason explicit. Python and TypeScript
already rejected the same input, so this closes a Go-only divergence.

The godoc that called this area "fail-closed, never fail-open" was accurate for
the newer-signer direction only, and the test it pointed at pinned the byte
equality that made the injection direction fail OPEN. Both are replaced: the
tests now build a clean and a tampered offer that render identically, at three
depths, so a rejection can only come from the unknown field itself. The paired
accessors also stated the pinned option set with different clause lists; they now
read identically.
…ng definition

The SDK guard is worth little if only this SDK applies it: a conforming verifier
written from the spec would still verify a signature over a message an
intermediary had appended unknown fields to. The rule now sits where the
canonical form is defined, on Offer.signature, so it binds every implementation
and both signed payloads rather than one language's helpers.

It states the requirement at every depth — a nested message and each element of a
repeated or map field carries its own unknown-field set — and names the reason:
proto-JSON emits only what the schema defines, so bytes reconstructed from such a
message omit part of what the signer covered. It also draws the line against the
extension mechanism, which is unaffected: ext and ext_critical are declared
fields inside the signed bytes, never undeclared field numbers.

Comments only — no field, message, or wire change. gen/ regenerated with the
pinned buf 1.66.1 (descriptor, Go types, and the Pydantic/Zod descriptions the
proto comments feed), and the changelog mirrored to the website. The reference
page renders the Offer table from the committed descriptor, so it picks the text
up without a hand edit.
…rpus

Reverting the empty-idempotency-key guard in the Python and TypeScript acceptance
payloads left every gate green in all three languages — it was the last omittable
field the corpus did not cover, so the omission rule could be dropped there
silently, re-opening the cross-language byte divergence the requester_id vector
was added to close.

TransactionRequest.idempotency_key carries min_len:1, so an empty key never
reaches the wire; these accessors sit below that check and must still agree on
the bytes, which is exactly why the gap was invisible.

The corpus gains one vector and nothing else: the existing entries and their
signatures are byte-identical, so no already-issued signature is affected. The
Python and TypeScript replay suites iterate the vector list, so they pick it up
with no per-vector code.
Go omits unpopulated fields structurally, through the pinned protojson option
set. The Python and TypeScript faces hand-build the payload, so they re-encoded
that one layer-wide rule as a conditional per key — and a missing conditional is
precisely how requester_id came to sign bytes Go never produces. Assembling the
whole record and filtering empty members once means a field added to
AgentAcceptancePayload cannot arrive without its omission.

Byte-neutral: the committed acceptance corpus replays unchanged in both
languages, which is the check that the refactor preserved every rendering.

Also corrects the TypeScript AcceptanceInput doc, which still described the
omission as applying to requesterDomain alone. That is the doc an editor surfaces
at the call site, so it outranks the function comment already fixed.
The test module restated the wire-versus-canonical contract in full, so the
correction to the retired camelCase description had to be made twice, in lockstep,
in the same change — the drift this kind of duplication causes, demonstrated on
itself. The description now lives only in ramp_sdk.wire_canon; the test docstring
keeps what the suite actually pins, plus the one consequence its assertions depend
on (from_wire_offer keeps the signature fields, so the comparisons strip them by
hand).

The same docstring also still announced the module as not yet existing, a red-step
narrative left standing after the module landed. Stated as history now, matching
the acceptance suite.
@legendko

Copy link
Copy Markdown
Collaborator Author

Update — unknown-field hardening + the last omission gap

A second review of the branch found a fail-open in offer verification. The fix is a
behaviour change to the Go SDK plus a normative rule in ramp.proto; everything above
still stands except the corpus count, corrected below.

The fail-open

An on-path party could append an unknown protobuf field to a signed Offer and
helpers.VerifyOffer still returned nil. protojson renders only fields the schema
defines, so the appended bytes never reached the canonical form — the signature kept
matching the fields Go could render, while the unauthenticated bytes rode along on a
message the recipient treats as signature-verified and survived a proto.Marshal
round-trip onward. That directly contradicts the guarantee stated on Offer.signature:
"an intermediary (Broker) cannot tamper … without invalidating it."

This branch did not introduce it. canonicalsign.go is untouched across every commit;
the behaviour is protojson semantics present identically on main. What the branch did
contribute was a godoc labelling the area "fail-closed, never fail-open" — true only for
the newer-signer direction — and a test that pinned the byte-equality making the injection
direction fail open. Both are replaced.

Python and TypeScript were already correct. Both canonicalizers keep unrecognized
members, so their bytes differ and verification fails. Go was the sole outlier; this closes
a cross-SDK divergence rather than opening one.

What changed

  • canonicalSignPayload refuses a message carrying unknown fields, at every depth.
    Unknown-field sets are per-message, so the walk visits nested messages and every element
    of a repeated or map field — a top-level-only check passes a payload whose tampering sits
    one level down. Written out explicitly rather than delegated to a traversal helper: it is
    a signature-coverage guard, so its reachability rules belong where they can be audited.
  • VerifyOffer surfaces the refusal as ErrOfferSignatureInvalid. This is load-bearing,
    not cosmetic: the downstream Exchange maps that sentinel to KindSignatureInvalid per the
    ADR-019 denial vocabulary and sends anything else to KindInternal. A raw sentinel would
    have reported a tampered offer as an internal fault. SignOffer and CanonicalOfferBytes
    return it directly — signing or persisting such a message is a caller bug.
  • The rule is normative, not just enforced. Offer.signature's canonical-signing block
    now states it, so a verifier written from the spec rejects too. Comments only — no field,
    message, or wire change — with gen/ regenerated under the pinned buf 1.66.1. The
    reference page renders the Offer table from the committed descriptor, so it picked the
    text up with no hand edit.
  • The last omission gap is pinned. Reverting the empty-idempotency_key guard in Python
    and TS left every gate green — it was the only omittable field the corpus did not cover.
    A second vector closes it.
  • Both ports now filter the assembled record once instead of guarding key by key. A
    missing per-key guard is exactly how requester_id came to sign bytes Go never produced.
    Byte-neutral: the corpus replays unchanged, which is the check.

No legitimate traffic regresses

An offer signed with a field this build cannot render already failed to verify, because
the renderer dropped it and the bytes differed. The refusal only makes the reason explicit.
The only newly-rejected messages carry unknown fields that were not signed — injections.
Extensions are unaffected: they ride in ext / ext_critical, declared fields inside the
signed bytes, never undeclared field numbers.

Downstream needs no change. It never signs or relays messages with unknown fields, and the
error kind it maps is preserved by design.

Corrections to the sections above

Above Now
"it gained the one empty_requester_id vector … 11 insertions, 0 deletions" Two appended vectors (empty_requester_id, empty_idempotency_key) — 22 insertions, 0 deletions. Still no re-signing; the pre-existing vectors and their signatures are byte-identical
Test counts (Python 520, TS 510) Python 526, TS 514 — the second vector fans out through the parametrized replays

Testing (re-run on the final tree)

  • go build ./... && go vet ./... && go test ./sdk/go/... ./conformance/... — green.
  • ./scripts/ci-local.shCI-local: PASS (buf pin, lint, generate, descriptor, gen drift,
    validation corpus, doc conformance, parity-matrix drift, SDK types export).
  • API-surface parity + parity-matrix — 11 passed.
  • sdk/python/tests — 526 passed. sdk/ts && npm test — 47 files / 514 tests; tsc clean.
  • website && npm run build — 79 pages; the new rule verified present in the rendered
    reference page.
  • Mutation-checked: with the walk crippled to the top level, every nested and
    repeated-element assertion fails. The fixtures pair a clean and a tampered offer that
    render identically, so a rejection can only come from the unknown field itself.

legendko added 3 commits July 24, 2026 18:34
…e rule

The rule as first written said a message carrying unknown fields MUST NOT be
canonicalized and a verifier MUST reject it. That binds implementations it should
not. Whether a canonicalizer can reproduce the signed bytes depends on what it
does with content it has no schema for:

  - OMITTING (proto-JSON emits only schema-defined fields): it cannot reproduce
    those bytes at all, so it must refuse the message rather than emit a reduced
    rendering, at every depth.
  - PRESERVING (carries unrecognized members through): it reproduces them
    faithfully and has nothing to refuse.

Either way an appended field cannot pass — the omitting canonicalizer refuses the
message, the preserving one renders the appended member into bytes the signer
never covered. Only the omitting case ever failed open, and that is the case the
Go guard closes. The Python and TS ports are preserving canonicalizers: they
already reject the tamper case on a byte mismatch, and they verify a message
whose signer covered the unknown member, which is the forward-compatible outcome
rather than a parity break. The changelog said they "already rejected such
input", true only of the tamper direction; it now states the distinction, as does
the Python module docstring that made the same claim.

The refusal is also now classifiable. CanonicalOfferBytes was exported for
out-of-package callers persisting evidence, and its doc promises the refusal, so
those callers need something to match on: errUnknownFields becomes
ErrUnknownFields, registered as a Go-idiomatic exclusion beside the other
errors.Is sentinels. VerifyOffer wraps BOTH it and ErrOfferSignatureInvalid, so
the denial mapping downstream keeps resolving through the signature sentinel
while a caller wanting the specific reason can ask for it.

The accessor doc no longer re-enumerates the depths the rule reaches — one copy
had already lost maps. That list lives in the normative comment and in the walk,
which is the audit point.

No verification outcome moves: proto comments only, regenerated with the pinned
buf, and the guard is untouched.
…field walk

Two branches of the walk had no test.

The map branch is reachable and guards the extension surface: Offer.ext is a
google.protobuf.Struct whose `fields` is a map with message values, so an unknown
field planted on a Struct Value sits two levels down and behind a map. Nothing
shallower catches it. Disabling the map branch fails this test and no other.

The entry branch is deliberately absent — the walk recurses into map values, not
into the synthetic entry messages carrying them. That is total only because
protobuf-go discards unknown bytes planted on an entry at unmarshal, so they never
reach the canonicalizer nor the wire again. The second test pins that assumption
and says what to do if a protobuf upgrade ever breaks it.

Also renames the newer-canonical-form test to say what it actually pins. It
asserts the rejection OUTCOME for that direction and passes with the guard
removed, since the plain signature mismatch yields the same sentinel; the guard is
gated by the injected-unknown tests, where clean and tampered render identically.
…tory

The design history said the acceptance payload is "the one place all three SDKs
hand-build the object", two sentences after correctly noting that Go inherits the
omission from protojson. Go renders an AgentAcceptancePayload message; only the
Python and TS ports hand-build. That distinction is the paragraph's own point.

Both ports claimed a future field "cannot arrive without" its omission. The filter
tests for the empty string, so a non-string field's zero value would pass it and
diverge from Go. The comment now says what holds: every member the message has is
a string, the field-set guard pins that list, and a non-string field would need
its own zero-value test.
@legendko
legendko merged commit 5321e3d into main Jul 24, 2026
4 checks passed
@legendko
legendko deleted the feat/sdk-canonical-offer-bytes-helper branch July 24, 2026 18:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant