Add libmacaroons V2 serialization formats and fix V1 byte-handling bugs - #35
Open
kandalf wants to merge 3 commits into
Open
Add libmacaroons V2 serialization formats and fix V1 byte-handling bugs#35kandalf wants to merge 3 commits into
kandalf wants to merge 3 commits into
Conversation
Three defects, each reachable from Macaroon.from_binary/from_json on caller-supplied input: 1. BinarySerializer sized its packets with String#length, so a multibyte identifier or caveat produced a length prefix smaller than the bytes actually emitted, corrupting the packet stream. deserialize had the matching problem, slicing the decoded stream by character. 2. A packet header claiming zero length left the read index unchanged, so deserialize spun forever instead of raising. Headers are now validated as four hex digits with a length that can at least cover the header and the trailing newline. 3. JsonSerializer#serialize used caveats.map!, mutating @Caveats in place and replacing each third-party caveat's verification_id with its base64 form. Calling serialize_json twice therefore double-encoded the vid and produced a macaroon that could not be verified. Also widens vid decoding on the JSON side. js-macaroon writes V1 JSON vids as url-safe unpadded base64, which Base64.strict_decode64 rejects outright; decoding now accepts either alphabet, padded or not, matching js-macaroon's base64ToBytes and libmacaroons' b64_pton. Output is unchanged — still standard and padded, which the stricter Go and Python readers require. Wire-facing values are funnelled through Utils.binary on the way in and Utils.readable on the way out, so lengths are always measured in bytes while identifiers and predicates still compare equal to ordinary UTF-8 Strings — Verifier#satisfy_exact depends on the latter. RawMacaroon.build lets the deserializers rebuild a macaroon from signed parts without inventing a placeholder key and discarding a derived signature. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ruby-macaroons has only ever spoken the V1 binary format (libmacaroons/v1.c)
and a V1 JSON encoding. js-macaroon speaks neither: it refuses to export V1
binary outright, so the two libraries have had no binary encoding in common
and only V1 JSON in common at all.
This adds the two V2 encodings from libmacaroons/v2.c:
BinaryV2Serializer a version byte followed by typed, uvarint-length
prefixed fields, with EOS markers closing the header,
each caveat and the caveat list.
JsonV2Serializer the same field set, each written as a plain string
under its short key, or as url-safe unpadded base64
under key + "64" when the bytes are not valid UTF-8.
No cryptographic changes are involved. HMAC-SHA256, the
macaroons-key-generator derived key, the keyedHash2 third-party
construction and the zero-key request binding are identical across
libmacaroons, js-macaroon and this library, and libmacaroons applies them
regardless of format. Only the framing differs.
Macaroon.from_serialized detects the encoding from the data, the way
macaroon_deserialize does in libmacaroons and importMacaroon does in
js-macaroon: V1 binary, V1 JSON, V2 binary or V2 JSON. It also unwraps a
base64 layer if one is present, since macaroons arriving over HTTP almost
always have one. from_binary and from_json remain as aliases, so callers
that used to try one and rescue into the other no longer need to.
New macaroons still default to version 1, so upgrading changes no bytes for
any existing caller. Version 2 is opt-in per macaroon and is required to
exchange binary macaroons with js-macaroon. A macaroon parsed from the wire
re-serializes in the version it arrived in, so a service can accept both and
answer in kind.
The V2 expectations in the specs are taken verbatim from js-macaroon's
test/serialize-binary.js and test/serialize-json.js, so they act as an
interoperability oracle rather than a record of whatever this library
happens to emit. Round trips through js-macaroon were verified in both
directions, including third-party caveat vids, which cross the boundary
intact because RbNaCl::SimpleBox and tweetnacl secretbox agree on
nonce-then-ciphertext layout.
BinarySerializer#serialize and #deserialize keep their existing contract:
serialize still returns base64, deserialize still accepts it. serialize_raw
exposes the wire bytes for callers that want to hand them to libmacaroons
directly, and deserialize now additionally accepts the raw packet stream —
the two forms are unambiguous, since packets open with four hex digits and
their base64 always opens "MDAx".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds a Serializing section covering the four encodings, the version: argument and its default, and the fact that Macaroon.from_serialized detects the format from the data. Calls out that js-macaroon cannot read or write V1 binary, since that is the detail which decides whether a caller needs version: 2. 1.1.0 rather than 2.0.0: the V2 formats are additive and the default version is unchanged, so no existing caller emits different bytes after upgrading. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the libmacaroons V2 binary and V2 JSON (
v2j) formats, plus automaticformat detection on deserialization, and fixes three pre-existing bugs in the V1
serializers.
The motivation is interoperability. ruby-macaroons has only ever spoken the V1
binary format and a V1 JSON encoding. js-macaroon
speaks neither — it refuses to export V1 binary outright — so the two libraries
have had no binary encoding in common, and only V1 JSON in common at all.
Anything with a browser client is currently forced into base64-wrapped JSON and a
rescueto guess which format arrived.Compatibility
Nothing an existing caller does changes.
Macaroon.new(..., version: 2), so upgrading emits identical bytes.BinarySerializer#serializestill returns base64;#deserializestill acceptsit.
#serialize_rawis new, for callers who want the wire bytes.JsonSerializeris unchanged in output, including the"vid":null,"cl":nullkeys.
Macaroon.from_binary/from_jsonaccept everything they did before, and nowalso V2, raw bytes, and base64-wrapped input. Strictly a superset.
Released as
1.1.0rather than2.0.0on that basis.What's added
BinaryV2Serializerlibmacaroons/v2.c)JsonV2Serializerkey + "64"when not valid UTF-8Macaroon.from_serializedNo cryptographic changes. HMAC-SHA256, the
macaroons-key-generatorderived key,the
keyedHash2third-party construction and the zero-key request binding areidentical across libmacaroons, js-macaroon and this library, and libmacaroons
applies them regardless of format. Only the framing differs.
Bug fixes (first commit, independent of V2)
7e879eastands alone and can be taken without the rest:BinarySerializersized packets withString#length. A multibyteidentifier produced a length prefix smaller than the bytes emitted, corrupting
the stream.
deserializehad the matching problem, slicing by character.advanced, so
from_binaryhung instead of raising — reachable fromcaller-supplied input.
JsonSerializer#serializeusedcaveats.map!, mutating@caveatsinplace and replacing each third-party
verification_idwith its base64 form.Calling
serialize_jsontwice double-encoded the vid and produced a macaroonthat could not be verified.
Also widens
viddecoding: js-macaroon writes V1 JSON vids as url-safe unpaddedbase64, which
Base64.strict_decode64rejects. Decoding now accepts eitheralphabet, padded or not. Output is unchanged — still standard and padded, which
the stricter Go and Python readers require.
Tests
spec/integration_spec.rbis untouched. New coverage lives inspec/serialization_spec.rb: 27 → 105 examples.The V1 and V2 expectations are copied verbatim from libmacaroons' own suite
(
test/unit/*.vtestandtest/unit/serialization_{1,2,3}). There is no macaroonspecification for serialization — the paper defines the cryptographic
construction and says nothing about wire formats — so libmacaroons is the de
facto definition and its vectors are the closest thing to a conformance suite
that exists.
Reproduced as literals, so no new dependencies and nothing outside the repo
is required to run the suite.
One divergence surfaced while doing this: V2 JSON with no caveats. libmacaroons
emits
"c":[], js-macaroon omits the key. Both readers accept either; thisfollows libmacaroons.
Round trips were additionally verified against js-macaroon in both directions,
including third-party caveat vids — those cross intact because
RbNaCl::SimpleBoxand tweetnacl
secretboxagree on nonce-then-ciphertext layout. That harnessneeds Node, so it is deliberately not part of the repo.
Note on CI
.travis.ymlis untouched and points at Ruby versions predating this work, so CIwill not run here. Happy to add a GitHub Actions workflow, in this PR or a
separate one, if that would be useful.
🤖 Generated with Claude Code