Skip to content

feat(peernet): share and import verified release bundles - #1069

Open
javi11 wants to merge 5 commits into
feat/peernet-metadata-sharingfrom
feat/peernet-bundles
Open

javi11 wants to merge 5 commits into
feat/peernet-metadata-sharingfrom
feat/peernet-bundles

Conversation

@javi11

@javi11 javi11 commented Nov 14, 2026

Copy link
Copy Markdown
Collaborator

Summary

Second PR of the peer network stack, on top of #1068. Adds exposure level 3 (share bundles): a node can serve a sanitised, compressed description of a finished import, and another node can mount the release from it after verifying it locally, without downloading or analysing the NZB.

Design: docs/superpowers/specs/2026-11-12-peer-network-design.md (the "Phase 3" sections)
User docs: docs/docs/3. Configuration/peer-network.md

Bundle format

  • Own versioned schema, canonical protobuf wire encoding, strict decoding (decode then re-encode must reproduce the bytes), zstd with bounded streaming decompression and bundle-wide budgets. Limits: 8 MB compressed, 64 MB decompressed, 500,000 segments, 10,000 files.
  • The Bundle ID is the hash of the canonical bytes and is carried in the signed release record, so a peer cannot substitute another bundle.
  • Never contains local or store paths, rclone password or salt, known holes, indexer, category, ARR or provider data.
  • Not shareable: rclone-encrypted files, files repaired by local PAR2 patches, and keyed releases unless the password came inside the NZB itself (recorded at import time; user-supplied and filename passwords do not count).

Serving and fetching

  • Served only on the mutual-TLS peer listener, only at level 3, only for bundles this node announced in that network, with exclusions re-checked at serve time. Rate, byte and concurrency limits per peer, per address and node-wide.
  • Bundles are rebuilt on demand; builds are deterministic.
  • Fetched only on demand from a local caller, only from the publisher itself, and verified against the IDs in the signed record.

Receiving (verified in every network before anything is written)

  • Hash and structure, no gap references, each file exactly one NZB entry in order, declared sizes against the NZB, article presence on the user's own providers, yEnc headers of randomly sampled articles against the layout, and a structural media check through the real reader stack that reads to the end of the file.
  • Only verified media plus subtitles and small NFO files are written; files a media server or the application could act on reject the bundle.
  • The import is playable at once. A full check of every segment is forced. It is announced to ARR applications and published to the network only after that check passes. While verifying, an ARR-submitted item stays in the SABnzbd-compatible queue as in progress; a failed or stale import is removed and reported as failed.
  • No archive parser and no key derivation run when a bundle is accepted.

Importer fast path

When an NZB is imported and a peer offers a bundle for the same release, the layout is taken from the bundle and archive analysis is skipped. On by default for private networks; for the open network it is off unless peernet.open_network.fast_path is enabled, and then only for bundles announced identically by two known publishers. It falls back to the normal import on any error.

Older library content

Imports in the older metadata format had a file ID but no release ID. They now get a release identity derived from their set of file IDs, computed from the existing index with no NZB parsing, and health evidence matches by file ID.

Stream listings

Releases offered by peers with a fetchable bundle are listed for a media ID, marked as coming from the peer network, and are fetched, verified and imported on play. Media IDs learned this way are not republished.

Changes outside the feature

  • internal/importer: optional fast-path hook after NZB parsing, password-origin recording, a hold flag for imports awaiting verification.
  • internal/health: optional gate that forces full checks for peer imports and reports their results.
  • internal/api: SABnzbd-compatible queue and history handling of held items, additive DTO fields, stream listing and play route.
  • Migrations 057-062 for SQLite and PostgreSQL.

Test plan

  • go build ./..., go vet ./...
  • go test -race ./...
  • internal/database tests against PostgreSQL 16, migrations applied from empty
  • golangci-lint clean on the new and touched packages (run per package)
  • Frontend: type check, unit tests, production build
  • Fuzzing of the bundle decoder, decompressor and validator
  • Multi-node tests on loopback: publish at level 3, fetch, verify, import, forced full check, announce once; failure variant
  • Two live instances: serve and import a real release over a private network
  • yEnc sampling cost and behaviour against real providers
  • ARR client behaviour while an item is verifying (reasoned from their status handling, not run)
  • Stream listing and play with a real client
  • Settings UI checked in a browser

Notes for reviewers

  • The structural media check accepts Matroska, MP4 (including fragmented) and transport streams; other containers get bounded reads through the file. A Matroska file with no seek index has its end checked only by size and readability.
  • Each accepted bundle costs up to 24 sampled articles of extra download.
  • Password-protected releases imported before this change, or while the feature was off, are not shareable as bundles.

Adds exposure level 3. A node can serve a sanitised, compressed description
of a finished import (a bundle) and another node can mount the release from
it after verifying it locally, without downloading or analysing the NZB.

Bundles:
- canonical, strictly decoded format with zstd compression, bounded
  decompression and bundle-wide budgets; fuzz tested
- never contains local or store paths, rclone secrets, known holes,
  indexer, category, ARR or provider data
- not shareable: rclone-encrypted files, files repaired by local PAR2
  patches, keyed releases unless the password came inside the NZB itself

Receiving:
- verified in every network before anything is written: hash against the
  signed record, structure, no gaps, one NZB entry per file, sizes against
  the NZB, article presence on the user's providers, sampled yEnc headers,
  and a structural media check through the layout
- playable at once; a full check of every segment is forced; the import is
  announced to ARR applications and published only after that check passes
- failed or stale imports are removed and reported as failed
- importer fast path for private networks; opt-in for the open network
  and then only for bundles announced identically by two known publishers

Also:
- older imports get a release identity derived from their file set, with
  no NZB parsing, and health evidence now matches by file
- stream listings show peer releases with a fetchable bundle, imported and
  verified on play
- settings UI for level 3, activity statistics and badges for peer imports
Second hardening round for bundle imports, from a re-review and a live
two-instance run.

- wire the article header reader into the bundle importer; without it
  every peer bundle import was rejected. The importer now refuses to be
  built with a required collaborator missing
- a pool entry may appear once per table, tables may share entries only
  when identical, and only the first and last segment of a table may be
  partial
- every table is sampled (first, last and a random article); bundles with
  too many tables to sample are rejected; the forced full check also
  verifies headers on a sample of articles
- the open network accepts only plain layouts in containers with a
  structural check; archive, keyed and clipped layouts stay private-only
- stricter fragmented MP4 tail, interior checks for Matroska and
  transport streams
- corroboration counts the address actually connected to, so several
  names for one server count once
- failure paths compare the file ID before removing anything
- a degraded or failed fast-path import falls back to a normal import
- offers are filtered before dialling; fetch limits and plain refusals
  are reported as such
- lower decode memory, one verification at a time, 16 MiB decoded cap
- catch-up publish when a network level is raised
…r checks

Removes the restriction that limited the open network to plain layouts in
structurally verifiable containers. Archive, keyed and clipped layouts and
all media containers are accepted on every network again.

- archive extents are bound to the archive's own file headers (RAR4, RAR5,
  7z with a plain end header): stored entry, data start, packed size and
  continuation flags must match the bundle; a contradiction is attributed
  to the publisher
- where headers cannot be read the file is accepted only with a real
  structural media check, never on readability alone
- the article sampling budget scales with release size instead of
  rejecting releases with many volumes; the last article of every table
  is always checked
- bounded header parsing with fuzz targets
- documentation for the wiring guard, fetch outcomes, catch-up publish
  and the network terminology
A file stored across several archive volumes is read as one slice per
volume. Bundles can now describe that layout explicitly, so the most
common release shape can be built, served and accepted.

- optional per-file part list in the bundle; absent means one posted
  file, so existing bundles keep their bytes and IDs
- partial ranges are allowed only at part boundaries; each part is a run
  inside one NZB entry
- each sliced part is bound to its volume's own file header (data start,
  packed size, continuation flags, RAR 5 volume order); unbound slices
  need a real structural media check
- article checks run per part, and the last article of every part is
  always checked
- slices not backed by an archive header stay rejected
Found by a live two-instance run and a review of the layout changes.

- bundle files are named after their virtual path; archive content had
  no name, so every archive release was announced and then rejected
- the publisher runs the receiver's file policy before announcing
- real-pipeline test: importer, builder, verify, accept and read-back for
  plain, generic-container, RAR4/RAR5 single and multi-volume and
  password-in-NZB releases
- parts after the first must start near their article's beginning and
  parts before the last must end near its end, which rejects files
  stitched from small windows
- the queue row id is resolved after an upsert, fixing history writes
  for direct peer imports
- local-policy rejections refund the per-publisher fetch budget, and the
  default budget suits browsing
- the importer is not installed when a required collaborator is missing
- catch-up publishes are counted; status reports header-bound parts
- current rules consolidated in the spec

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant