feat(NET-1180): diverge worker/portal assignment wire formats - #214
Merged
Conversation
Adds two new flatbuffers root types alongside the legacy Assignment: - WorkerAssignment: dataset_base_url, schema_id, tables_present per chunk; worker entries keep encrypted_headers. No files/base_url/last_block_hash (the worker resolves its file set from the schema, not a wire-provided list; base_url was always redundant with the chunk id). - PortalAssignment: block range, last_block_hash, last_block_timestamp per chunk; a dataset-level schema_id reference for query validation. No download/auth fields at all. Renamed the existing per-worker-entry table WorkerAssignment -> WorkerEntry to free the name for the new worker-facing root (pure rename, no external crate references it directly; safe for the always-compiled legacy path). New types are gated behind the existing (previously unused) mvcc-chunks feature, matching common.rs's existing pattern for the NetworkState pointer fields -- verified compiling, testing, and clippy-clean both with and without the feature. Also: regenerating codegen with a flatbuffers-version-matched flatc surfaced that deprecated fields are excluded from generated *Args structs now, which required dropping the legacy per-worker `chunks` field write in AssignmentBuilder (already unused on the read side; iter_chunks uses worker_indexes) and regenerating the test fixture accordingly. See docs/assignment-wire-format.md in network-scheduler for the full field-by-field rationale.
This was referenced Jul 29, 2026
Contributor
|
Hm, I think flatbuffer file is missing |
…n schema files Addresses define-null's PR review comment -- the new root types lived inline in assignment.fbs instead of getting their own worker_assignment.fbs/portal_assignment.fbs. worker_assignment.fbs and portal_assignment.fbs now `include "assignment.fbs"` to reuse WorkerStatus/WorkerId/EncryptedHeaders/WorkerEntry, each generating their own *_generated.rs (flatc emits one file per schema, cross-referencing shared types via a hardcoded `use crate::<name>_generated::*`). assignment_fb.rs is now a thin facade re-exporting all three generated modules into one flat namespace, so builder.rs/reader.rs keep referencing `assignment_fb::Foo` unchanged -- purely a schema/codegen reorganization, no wire format or public API change. Also: narrowing #![allow(clippy::all, ...)] to just the generated-code modules (it was previously blanket-suppressing lints across all of assignment_fb.rs, including hand-written code) surfaced one genuine pre-existing nit -- a redundant `.clone()` on `PeerId` (which is `Copy`) in a test -- now fixed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
Author
|
Good catch — split into |
define-null
reviewed
Jul 30, 2026
define-null
reviewed
Jul 30, 2026
Contributor
|
It would be valuable to validate how worker_assignment and portal_assignment in terms of overall data compare with the legacy assignment. Can you maybe fetch the most recent assignment for the mainnet and get some numbers how large the portal/worker assignment would be? |
… dataset level Per define-null's PR review: last_block_hash is only ever meaningful as the dataset's head hash (its last chunk's value) -- confirmed nothing reads it per-chunk anywhere in the codebase, unlike last_block_timestamp which find_chunk_by_timestamp's binary search genuinely needs on every chunk. Moving it to PortalAssignmentDataset removes the hand-written derive-from-last-chunk helper in favor of a plain generated field accessor. PortalAssignmentBuilder::finish_dataset() now takes last_block_hash as a parameter instead of PortalAssignmentChunkBuilder having a per-chunk setter. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
Author
|
Ran this against the current mainnet assignment (235 datasets, 2031 workers). I derived
(Raw and in-memory size are the same number here — FlatBuffers is zero-copy, so there's no separate expanded representation. Legacy gzip is my own re-compression at the same settings as the other two, for a fair comparison, rather than the live download's exact Content-Length.) So each individual consumer downloads ~13-19% less than today. But since the legacy blob served both workers and portals from one buffer, fields shared by both (id, dataset_id, worker_indexes, the block-range key) were stored once; splitting means those now get duplicated across two blobs. Net effect: total data the scheduler generates/stores/publishes goes up by roughly 1.7x, even though per-consumer traffic goes down.
|
define-null
reviewed
Aug 3, 2026
define-null
reviewed
Aug 3, 2026
…hemas Per define-null's PR review -- the module name and its original legacy-only contents made it non-obvious that it now also houses re-exports and hand-written impls for the worker/portal split types. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
define-null
approved these changes
Aug 3, 2026
toschoosqd
added a commit
to subsquid/worker-rs
that referenced
this pull request
Aug 5, 2026
## Summary Adds discovery and decoding support for the new `WorkerAssignment` type (from NET-1180), without touching real chunk-serving. Scope is deliberately narrow — infra only, not a switch-over. - `visible_assignment()` now prefers `NetworkState.worker_assignment`, falling back to the legacy `assignment` pointer — mirrors the pattern already used in sqd-portal (PR #125), gated behind `mvcc-chunks`. - `fetch_worker_assignment()` decodes `sqd_assignments::WorkerAssignment` alongside the untouched legacy `fetch_assignment()`; shared gzip-download logic factored out into `download_gzipped()`. - In `p2p.rs`: when an update is sourced from `worker_assignment`, it's decoded and logged only — never fed to the legacy parser. `DatasetsIndex` and actual chunk-serving stay on the legacy path entirely. ## Notes - **Out of scope**: switching real chunk-serving over to the new format. `WorkerAssignmentChunk` has no `files`/`base_url` — deriving them from `schema_id`/`tables_present` needs the schema delivery mechanism, which is separate, not-yet-scoped work (tracked against NET-1180's design doc, not this ticket). - Temporarily pins `sqd-assignments` to NET-1180's branch commit (`1e334ec`) in both `[dependencies]` and `[dev-dependencies]`, since the new types aren't on sqd-network's master yet. Marked `# TEMPORARY` — needs re-pinning to master once [sqd-network#214](subsquid/sqd-network#214) merges. ## Test plan - [x] `cargo check` / `test` / `clippy -D warnings` / `fmt --check` clean, both with and without `mvcc-chunks` - [x] Full suite (lib + `e2e` + `query_concurrency` + `query_surface`), 97 tests, 0 failures 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Splits the legacy shared
Assignmentflatbuffers type into two purpose-built root types, gated behind the existingmvcc-chunksfeature (previously declared but unused):WorkerAssignment— what a worker needs to serve chunks:dataset_base_url,schema_id,tables_presentper chunk; worker entries keepencrypted_headers. Dropsfiles/base_url/last_block_hash(workers resolve their file set from the schema, not a wire-provided list;base_urlwas already redundant with the chunk id).PortalAssignment— what a portal needs for query routing: block range,last_block_hash,last_block_timestampper chunk; a dataset-levelschema_idreference for query validation. No download/auth fields at all.The existing per-worker-entry table (also named
WorkerAssignment) is renamed toWorkerEntryto free the name for the new worker-facing root. Pure rename — no external crate references it directly, and it's on the always-compiled legacy path so it can't be feature-gated.Full field-by-field rationale:
docs/assignment-wire-format.mdinnetwork-scheduler.Notes
mvcc-chunks; without the feature, behavior is byte-for-byte unchanged.flatcsurfaced that deprecated fields are now excluded from generated*Argsstructs, which required dropping the legacy per-workerchunksfield write inAssignmentBuilder(already dead on the read side —iter_chunksusesworker_indexes) and regenerating the test fixture accordingly. Unrelated to the schema split itself, but included since it was a compile-blocking side effect of the flatc bump.worker_assignment/portal_assignmentpointers and the confirmation/quorum loop — is intentionally not part of this PR; that's Vasilii's piece.Test plan
cargo test— new round-trip tests for bothWorkerAssignmentandPortalAssignment(crates/assignments/tests/test_split_assignments.rs)cargo check/cargo clippy -D warningsclean with and withoutmvcc-chunksAssignmenttests still pass against the regenerated fixture🤖 Generated with Claude Code