Repository navigation
Conversation
Adds exposure level 3. A node can serve a sanitised, compressed description of a finished import (a bundle) and another node can mount the release from it after verifying it locally, without downloading or analysing the NZB. Bundles: - canonical, strictly decoded format with zstd compression, bounded decompression and bundle-wide budgets; fuzz tested - never contains local or store paths, rclone secrets, known holes, indexer, category, ARR or provider data - not shareable: rclone-encrypted files, files repaired by local PAR2 patches, keyed releases unless the password came inside the NZB itself Receiving: - verified in every network before anything is written: hash against the signed record, structure, no gaps, one NZB entry per file, sizes against the NZB, article presence on the user's providers, sampled yEnc headers, and a structural media check through the layout - playable at once; a full check of every segment is forced; the import is announced to ARR applications and published only after that check passes - failed or stale imports are removed and reported as failed - importer fast path for private networks; opt-in for the open network and then only for bundles announced identically by two known publishers Also: - older imports get a release identity derived from their file set, with no NZB parsing, and health evidence now matches by file - stream listings show peer releases with a fetchable bundle, imported and verified on play - settings UI for level 3, activity statistics and badges for peer imports
Second hardening round for bundle imports, from a re-review and a live two-instance run. - wire the article header reader into the bundle importer; without it every peer bundle import was rejected. The importer now refuses to be built with a required collaborator missing - a pool entry may appear once per table, tables may share entries only when identical, and only the first and last segment of a table may be partial - every table is sampled (first, last and a random article); bundles with too many tables to sample are rejected; the forced full check also verifies headers on a sample of articles - the open network accepts only plain layouts in containers with a structural check; archive, keyed and clipped layouts stay private-only - stricter fragmented MP4 tail, interior checks for Matroska and transport streams - corroboration counts the address actually connected to, so several names for one server count once - failure paths compare the file ID before removing anything - a degraded or failed fast-path import falls back to a normal import - offers are filtered before dialling; fetch limits and plain refusals are reported as such - lower decode memory, one verification at a time, 16 MiB decoded cap - catch-up publish when a network level is raised
…r checks Removes the restriction that limited the open network to plain layouts in structurally verifiable containers. Archive, keyed and clipped layouts and all media containers are accepted on every network again. - archive extents are bound to the archive's own file headers (RAR4, RAR5, 7z with a plain end header): stored entry, data start, packed size and continuation flags must match the bundle; a contradiction is attributed to the publisher - where headers cannot be read the file is accepted only with a real structural media check, never on readability alone - the article sampling budget scales with release size instead of rejecting releases with many volumes; the last article of every table is always checked - bounded header parsing with fuzz targets - documentation for the wiring guard, fetch outcomes, catch-up publish and the network terminology
A file stored across several archive volumes is read as one slice per volume. Bundles can now describe that layout explicitly, so the most common release shape can be built, served and accepted. - optional per-file part list in the bundle; absent means one posted file, so existing bundles keep their bytes and IDs - partial ranges are allowed only at part boundaries; each part is a run inside one NZB entry - each sliced part is bound to its volume's own file header (data start, packed size, continuation flags, RAR 5 volume order); unbound slices need a real structural media check - article checks run per part, and the last article of every part is always checked - slices not backed by an archive header stay rejected
Found by a live two-instance run and a review of the layout changes. - bundle files are named after their virtual path; archive content had no name, so every archive release was announced and then rejected - the publisher runs the receiver's file policy before announcing - real-pipeline test: importer, builder, verify, accept and read-back for plain, generic-container, RAR4/RAR5 single and multi-volume and password-in-NZB releases - parts after the first must start near their article's beginning and parts before the last must end near its end, which rejects files stitched from small windows - the queue row id is resolved after an upsert, fixing history writes for direct peer imports - local-policy rejections refund the per-publisher fetch budget, and the default budget suits browsing - the importer is not installed when a required collaborator is missing - catch-up publishes are counted; status reports header-bound parts - current rules consolidated in the spec
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Second PR of the peer network stack, on top of #1068. Adds exposure level 3 (share bundles): a node can serve a sanitised, compressed description of a finished import, and another node can mount the release from it after verifying it locally, without downloading or analysing the NZB.
Design:
docs/superpowers/specs/2026-11-12-peer-network-design.md(the "Phase 3" sections)User docs:
docs/docs/3. Configuration/peer-network.mdBundle format
Serving and fetching
Receiving (verified in every network before anything is written)
Importer fast path
When an NZB is imported and a peer offers a bundle for the same release, the layout is taken from the bundle and archive analysis is skipped. On by default for private networks; for the open network it is off unless
peernet.open_network.fast_pathis enabled, and then only for bundles announced identically by two known publishers. It falls back to the normal import on any error.Older library content
Imports in the older metadata format had a file ID but no release ID. They now get a release identity derived from their set of file IDs, computed from the existing index with no NZB parsing, and health evidence matches by file ID.
Stream listings
Releases offered by peers with a fetchable bundle are listed for a media ID, marked as coming from the peer network, and are fetched, verified and imported on play. Media IDs learned this way are not republished.
Changes outside the feature
internal/importer: optional fast-path hook after NZB parsing, password-origin recording, a hold flag for imports awaiting verification.internal/health: optional gate that forces full checks for peer imports and reports their results.internal/api: SABnzbd-compatible queue and history handling of held items, additive DTO fields, stream listing and play route.Test plan
go build ./...,go vet ./...go test -race ./...internal/databasetests against PostgreSQL 16, migrations applied from emptygolangci-lintclean on the new and touched packages (run per package)Notes for reviewers