Repository navigation
Conversation
This was referenced Oct 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Packing enough small chunks could create a xorb that Crab immediately rejects on read. The 10,000-file protected-receive qualification reproduced
xorb contains 10000 chunks; maximum is 8192with real recipes and payloads.Seal the current xorb before accepting a new unique chunk beyond the locked Xet chunk-count limit. The check lives beside the existing decoded-size rollover in the common state-mutating builder path. Deduplication still runs first; sealing still requires the existing materialization admission callback before mutation. Run continuity cannot override a format limit.
Stacked on #579 (
codex/selected-publication-catalog). This PR contains only the Xet builder, its regression tests and the crate README; the protected-consumer changes are a separate follow-up. No dependency, format, parser-limit or configuration changes.Validation:
cargo clippy --no-deps -p crab-xet --no-default-features -- -D warningspasses.chunker,upload-concurrencyenabled. The dependent protected-receive integration in fix(auth): isolate capsule verification and page catalogs #583 passes all 81 receive tests, including four real 100/10,000-file run/checkpoint cases that exercise splitting at the chunk limit. Deployed/S3 and 10 GiB qualification remain separate gates.Evidence map:
XorbBuilder::push_precompressed_with_reserve_and_admissionincrates/crab-xet/src/xorb/builder.rs.pack_decoded_chunk_groupincrab/src/git/push.rs; direct prepared staging incrates/crab-staging/src/stream.rs.finalize_current_after_admissionretains writer admission and serialized-payload pool ordering.xet-core-structures1.6.0 exportsMAX_XORB_CHUNKS = 8 * 1024; its raw-xorb constructor asserts the same maximum. Crab's parser rejects larger chunk counts before allocating metadata.origin/mainatcc2e4e25f46eeaee3cfd30025d257397c17a6049has the same builder function without the count rollover; the original builder file blob matches this PR's base.Is this the best fix? Enforcing the existing format limit at the common ingestion boundary covers ordinary push and prepared staging together, preserves dedup/admission ownership, and avoids increasing reader limits or adding path-specific packing logic. Production change is four net lines; the larger diff is regression coverage.
Already-written oversized xorbs remain invalid and are still rejected; this change prevents new invalid output and does not silently repair stored data. Full installed-CLI/10 GiB and hosted-S3 performance qualification remain separate gates.