Repository navigation
sv2: validate declared custom jobs through the template provider (ProposeTemplate) - #976
Draft
average-gary wants to merge 19 commits into
Draft
average-gary wants to merge 19 commits into
average-gary wants to merge 19 commits into
Conversation
Wire layer for the TDP job-validation extension (docs/sv2-job-validation-draft.md §3-5): the four ValidateCustomJob messages (0x77-0x7a) as binary_sv2 structs, and SetupConnection flag bit 0 (REQUIRES_JOB_VALIDATION). Setup now accepts that bit and echoes it in SetupConnection.Success; any other set bit is still rejected with unsupported-feature-flags. The messages are `pub` because the rbitcoin-test journey decodes TDP frames cross-crate (as it does the SRI types today) and they have no in-crate reader yet; the dispatch handlers follow in the next step. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
ValidateCustomJob (docs/sv2-job-validation-draft.md §4.1) carries a full wtxid list and the missing transactions, so it cannot fit the 64 KiB cap sized for SubmitSolution. The reader now bounds every client frame at the ValidateCustomJob maximum while reading (40 fixed bytes, a B064K coinbase, 65535 wtxids, and a block's worth of B016M txs: 6_359_306 bytes), then applies the per-type cap once the frame is whole, so every other client message still closes the session past 65557 bytes. codec_sv2 7.0.0 decrypts the header into a private buffer and never exposes it mid-frame, so the type is only known with the whole frame. Seeing it earlier would mean replacing NoiseDecoder's transport path with our own chunked decrypt loop; the per-session RAM bound is the same either way, since a client can always pick 0x77. Named RAM trade: one in-flight client frame of ≤ ~6.4 MB per session, ≤ ~51 MB at MAX_SESSIONS. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
docs/sv2-job-validation-draft.md §4.1–4.3: on a session that negotiated
REQUIRES_JOB_VALIDATION, a ValidateCustomJob whose wtxids are all in the
mempool is resolved, assembled under a header on the tip (time ≥ MTP + 1,
expected nBits, nonce 0, merkle root over the placeholder coinbase and the
declared txids) and run through ChainHub::check_block_proposal. Ok(fees)
retains the job under the next template id, exactly like a built template,
and answers Success{request_id, template_id, fees}; a reject string answers
Error{error_code}. Without the flag the message is still ignored.
The tip read (height, prev hash, time, bits) moves out of template::build
into template::next_header so both paths share it, and Template splits
into the NewTemplate coinbase split over a Job: what a session retains
under a template id for SubmitSolution and RequestTransactionData, so a
validated job carries no dead NewTemplate fields. CPU trade: one full
proposal check per request on the blocking pool (no scripts, no PoW).
A wtxid not in the mempool still drops the request; MissingTransactions
follows in the next step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
docs/sv2-job-validation-draft.md §4.1–4.2: each declared wtxid is resolved
against transaction_list first, then the mempool. Any still unknown answer
ValidateCustomJob.MissingTransactions{request_id, positions} with the
0-indexed positions in wtxid_list, so a JDS relays them as
ProvideMissingTransactions unchanged. The TP keeps no state across the
round trip: the second request repeats the job with the txs filled in, and
the supplied txs are retained with the job in block order so SubmitSolution
assembles the same block the check passed.
A supplied tx is matched to its slot by sha256d of the bytes as sent (the
wtxid of that serialization), so no transaction is decoded before it is
known to be declared; a declared blob that does not decode is
bad-missing-tx. The step-4 placeholder (close the session on an unknown
wtxid) is gone.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
docs/sv2-job-validation-draft.md §4.1 and §4.4: everything in a ValidateCustomJob comes from a JDC, so the draft's own codes run in order before any mempool lookup or transaction decode: the node in IBD answers job-validation-unavailable (the same gate that holds templates), a prev_hash off the tip stale-prevhash, a repeated wtxid duplicate-wtxid, a supplied transaction whose hash is not in wtxid_list bad-missing-tx (also a declared blob that does not decode), an undecodable coinbase bad-cb-decode, and only then the proposal check's own reject string (bad-cb-amount for an overpaying coinbase). A 32-byte wtxid is never amplified into a copy the job did not declare. Nothing is retained and no template id is taken on an error: the next id still answers template-id-not-found. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
docs/sv2-job-validation-draft.md §4.5: a SubmitSolution on the template id from ValidateCustomJob.Success is assembled from the retained job and the JDS's final coinbase (the extranonce where the placeholder stood), checked against the job's target, and accepted through ChainHub::accept_block like one for a pushed template; the tip advances to it. No production change: a validated job is retained as the same Job a built template is, so on_submit_solution already serves it. The journey pins that. first_template takes the setup flags and connects through connect_tp, so the solution journeys share one session bootstrap. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
docs/sv2-template-provider.md gains Plan D (custom job validation over TDP) with D1–D7 as executed, the named RAM and CPU trades, and its risks (no script execution on supplied txs, no operator opt-out, Success.flags echo, retention shared with templates). The plan table, the SetupConnection flag constraint, the per-type client frame cap, and Out of scope (the JDS role stays out; its node backend is served) are brought current. The draft spec is copied unchanged as docs/sv2-job-validation.md and owns the wire contract until it lands upstream (docs/README.md row); the operator doc, the COMPAT row, and a changelog.d fragment name the flag and what it enables, plus getblocktemplate proposal mode now answering bad-cb-amount. messages is crate-private: no other crate reads it, and crate pub is the cross-crate graph only. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The RFC in sv2-spec discussion reardencode#239 (plebhash, 2026-10-06) supersedes issue reardencode#217 and names the TDP message `ProposeTemplate`; its replies are `.MissingTransactions`, `.Success`, and `.Error`. Pure rename of the structs, message-type consts, the frame cap, the session handler, the journey names, log strings, the spec draft's message names, Plan D, the operator doc, the COMPAT row, and the changelog fragment. No behaviour change; the field changes the RFC asks for follow in the next commit. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…the validated tip sv2-spec discussion reardencode#239: `ProposeTemplate` carries what a JDS already has from `DeclareMiningJob` and nothing it would have to invent: `request_id`, `version`, `coinbase_tx_prefix`, `coinbase_tx_suffix`, `wtxid_list`, `excess_data` (opaque to this TP), plus our `transaction_list`. There is no `prev_hash` in the request (`DeclareMiningJob` has none); the TP validates on its own current tip and `ProposeTemplate.Success` names that tip as `prev_hash` next to `template_id` and `fees`, so the JDS can match it to `PushSolution.prev_hash`. The TP now builds the placeholder coinbase. `job::extranonce_len` parses the prefix (version, BIP144 marker and flag when present, an input count that MUST be 1, the prevout, the scriptSig length L and the P bytes present; 2 ≤ L ≤ 100, P ≤ L) and the coinbase is prefix ‖ zeros(L − P) ‖ suffix, decoded as before. A prefix that does not parse, or a result that does not decode, answers `bad-cb-decode`. `stale-prevhash` is gone: a declaration from before the tip moved fails the proposal check on its own (`bad-cb-height` for its BIP34 push), which the ordered-rejections journey now pins, together with a prefix carrying more scriptSig bytes than its length declares, a two-input prefix, and a truncated suffix. Every `Success` is checked against the tip. The untrusted-input order is unchanged: duplicate-wtxid, bad-missing-tx, coinbase assembly and decode, resolution, then the proposal check. `MAX_PROPOSE_TEMPLATE_PAYLOAD` is re-priced for three `B064K` fields (~6.5 MB per in-flight frame, ~52 MB at `MAX_SESSIONS`). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
docs/sv2-job-validation.md now names discussion reardencode#239 as the venue (reardencode#217 stays as the origin): §2 flow and the no-`prev_hash` rationale, §4.1 field table with `coinbase_tx_prefix` / `coinbase_tx_suffix` / `excess_data` and the placeholder derivation moved to the server side (E = L − P, the bounds, `bad-cb-decode`), the stale-prevhash rule replaced by how a stale declaration surfaces, §4.3 `prev_hash`, §4.4 table without `stale-prevhash` and with `bad-cb-decode`, §4.5 and §7.1 JDS mapping (relay the DeclareMiningJob fields unchanged; keep `Success.prev_hash` for the `PushSolution.prev_hash` check), §6 and §8 with reardencode#239. Plan D gains D8 and D9 for the rename and the field change, the frame cap is re-priced (~6.5 MB / ~52 MB), and the extranonce-tail assumption is listed as a TP-side risk. The changelog fragment, operator doc, COMPAT row, and docs index follow the wire. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…lates
`ProposeTemplate.Success` is `{request_id, template_id, prev_hash}`.
Bitcoin Core's IPC cannot produce a fee total for an externally proposed
block: `checkBlock` returns only reason, debug and result, and a
`TxCollection.makeTemplate` template throws on `getTxFees`, so sv2-tp
would have to send 0. One field with two meanings across TPs is worse
than none. The node's `bad-cb-amount` check already bounds the coinbase
at subsidy + fees and the Pool reads the claimed value from the coinbase
itself. `ChainHub::check_block_proposal` keeps returning `Ok(fees)`: it
is cross-crate API from D1 and GBT proposal mode may read it.
Retention: a JDS multiplexes many JDCs over one TDP connection, so
"keep the latest validated job" is the wrong guarantee. The draft §4.3
now requires the template rule instead (stale grace after a tip change,
oldest-first eviction by a per-connection cap of at least 64), notes
that `template_id` is unique per connection, and §7.1 says the JDS
needs no fallback because JDP 6.4.9 already has JDC propagate. The Plan
D eviction risk becomes that rule (D10). Agreed with the session that
owns Plan C.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Plan D lands on top of sv2/plan-c-fee-push (reardencode reardencode#949, reardencode#951), which supplies the Arc<Transaction> template bodies and the 64-slot same-tip retention that validated jobs now rely on instead of a pin. Readers coming from sv2-spec discussion reardencode#239 also get the pointer to the Core-IPC implementation of the same messages (sv2-tp reardencode#137). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`ProposeTemplate.Success` is `{request_id, template_id, prev_hash, fees}`;
`fees` is the sum of the fees of the declared transactions as
`ChainHub::check_block_proposal` computed it. This reverses 7f202e6's
field removal.
Only the validating node can derive the total: the fee of each
transaction is its inputs minus its outputs, which needs the UTXO set,
and the node computes it anyway for `bad-cb-amount`. Without it the Pool
has only the value the coinbase claims, which that check makes a lower
bound on the real total, not the total itself. Bitcoin Core's IPC does
not expose it today (`checkBlock` returns only reason, debug and result;
a `TxCollection.makeTemplate` template throws on `getTxFees`); that gap
is raised on bitcoin/bitcoin#35671 rather than designed around.
Red was E0560/E0609 (no field `fees`) on the round trip and
`expect_job_success`. Draft §2, §4.3, §6 and §7.1, Plan D (D4, D10) and
the changelog fragment say the field is returned.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`on_frame` awaited a ProposeTemplate's spawn_blocking inline inside the session select!, so one validation at a time and every other frame on that connection waited behind it, SubmitSolution for a found block included, along with the tip push, fee poll, and retire branches. A JDS multiplexes many JDCs over one TDP connection, so a block solution could sit behind someone else's declaration. The session now decodes the proposal and runs the chain-free prechecks (duplicate-wtxid, bad-missing-tx) on arrival, answering at once on failure, and queues the payload for a JoinSet of spawn_blocking + BlockingRegion validations, at most MAX_INFLIGHT_VALIDATIONS (4) per session, the rest in arrival order. A new select! arm takes each verdict and goes through the one reply path, retaining a valid job when its Success is sent; dropping the session aborts the set. job::validate stays complete on its own. Journey: 1200 supplied spends make the validation the costly frame; a RequestTransactionData and a SubmitSolution sent behind it are answered, and the solved tip's NewTemplate arrives, before the proposal reply, which then takes the next id and names the tip it started on. The request, with no disk behind it, must be answered in under a quarter of the validation it overlapped. Red was Success as the first frame. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
check_block_proposal re-decodes a parent for every input that spends it, which is what makes the D11 journey's 1200-input proposal slow. Record the cache follow-up and that the journey's margin currently depends on the inefficiency. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The D11 journey ordered a SubmitSolution's NewTemplate before a 1200-input proposal's Success. That margin came from the per-input parent decode in the proposal check, which reardencode#961 and reardencode#968 removed: the validation now takes about 210 ms at 1200 inputs and the accept's tip write races it, so the journey failed 10 of 10 runs on the merged check (Success first, or inconclusive-not-best-prevblk when the tip moved before the check). Pin the contract on the frame that has no blocking work behind it: a RequestTransactionData sent behind a 400-input proposal is answered before that proposal's Success and in under a quarter of its wall. A SubmitSolution sent behind a third copy is accepted and pushes the solved tip's template; it is not ordered against that copy's reply, which is the straddle the plan already names (Success on the starting tip, or the proposal check's reject once the tip moved). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Plan D no longer stacks on sv2/plan-c-fee-push (merged as reardencode#949 and reardencode#951) and its D1 step is upstream: reardencode#959 moved the proposal check onto ChainHub, reardencode#963, reardencode#967, reardencode#968, and reardencode#969 made it complete, one decode per parent, bounded, and right about spent coins after a reorg. Name those as the prerequisites and why: the check is the job-validation hot path. Drop the two risks they closed (no script execution, reardencode#967; the per-input parent decode, reardencode#968), name both ends of the tip-change straddle the D11 journey now accepts, fix the "no scripts" CPU-trade lines, and remove the getblocktemplate bad-cb-amount changelog entry that reardencode#959 already carries. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
sv2-spec#239 settled on the JDP shape for the missing-transactions leg (Sjors, 2026-10-08): the TP asks with a proper message and the JDS answers with one, reusing the 6.4.7 and 6.4.8 layouts under TDP numbers, and the TP holds the pending proposal. ProposeTemplate drops transaction_list; ProposeTemplate.MissingTransactions becomes ProvideMissingTransactions (0x78, TP to client) and ProvideMissingTransactions.Success (0x7b, client to TP) carries the txs. A JDS copies both payloads between the TP and its JDC unchanged. The session keeps a proposal it answered ProvideMissingTransactions under its request_id (wtxids and coinbase split, no transactions) in a table of at most MAX_PENDING_PROPOSALS (8) for provide_timeout (30 s, Sv2TpConfig like setup_timeout); past the bound the oldest is dropped. A provide for an id it does not hold, already consumed, or expired is ProposeTemplate.Error unknown-request-id; a proposal under an id still queued, in flight, or held is duplicate-request-id; a provide short of a requested position, with a tx the TP did not ask for, or with one that does not decode is bad-missing-tx and ends that exchange, the first two before any decode. These edge rules match sv2-tp reardencode#137. A validation that still lacks a transaction (one left the mempool between rounds) asks again for every position the mempool does not hold, the supplied ones included, since the TP holds no transactions across the round trip. The client frame cap moves the block-sized bound onto ProvideMissingTransactions.Success (~4.2 MB); ProposeTemplate drops to wtxids plus coinbase (~2.3 MB). Red was the compile failure of the journeys against the one-message shape (no pending table, no 0x7b handler; the old session ignored 0x7b), then a short provide answered ProvideMissingTransactions again and a ProposeTemplate under an in-flight id validated twice. The missing-transactions journey walks the round trip, each code, the bound, and expiry under a 1 s provide_timeout; the D11 journey completes each copy through its provide and refuses the in-flight duplicate. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The draft now specifies the shape sv2-spec#239 converged on: no transaction_list on ProposeTemplate; ProvideMissingTransactions (0x78) and ProvideMissingTransactions.Success (0x7b) reuse JDP 6.4.7 and 6.4.8 under TDP numbers (ours to propose; the thread has assigned none); the TP holds a proposal awaiting its provide for at least N seconds and may bound the holds per connection; unknown-request-id and duplicate-request-id join the error table; the Success table lists request_id, template_id, prev_hash, fees in wire order (the fees row had drifted to the top of the file). Section 6 replaces the stateless rationale with why the TP holds state (Sjors's request/provide point; the TxCollection handle is that state) and opens the queue-or-refuse question at the in-flight bound where rbitcoin and sv2-tp differ. Section 7.1 maps the JDS relay (payload copied, message type changed) and 8 records the 2026-10-08 turn. Plan D gains D12 for the round trip and the changelog fragment names the provide pair. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
average-gary
force-pushed
the
sv2/job-validation
branch
from
October 9, 2026 16:30
987f397 to
6cd50e7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft on purpose. The wire format this implements is still being worked out in sv2-spec discussion #239 (plebhash's RFC for a TDP message that lets a Job Declarator Server validate a custom job through a Template Provider). I'll keep this branch rebased on
masterand tracking the discussion until it settles, then mark it ready. If this isn't something rbitcoin wants to carry, close it; no hard feelings, the branch stays on my fork either way.What
A JDS running in Full-Template mode connects to rbitcoin's SV2 Template Provider, sets
SetupConnectionflag bit 0 (REQUIRES_JOB_VALIDATION), and can then:ProposeTemplate(0x77) carrying theDeclareMiningJobsubset (version,coinbase_tx_prefix/suffix,wtxid_list,excess_data);ProvideMissingTransactions(0x78, JDP 6.4.7 layout) for wtxids the node lacks, relay it to the JDC unchanged, and send the JDC's answer back asProvideMissingTransactions.Success(0x7b, JDP 6.4.8 layout);ProposeTemplate.Success(0x79:request_id,template_id,prev_hash,fees) orProposeTemplate.Error(0x7a, Core-style reject reason);SubmitSolutionagainst thattemplate_id; the job is retained under the same rules as pushed templates.The TP builds the placeholder coinbase itself (extranonce gap = scriptSig length from the prefix minus the bytes present), validates the assembled block through
ChainHub::check_block_proposal(everything except PoW and merkle root, no policy rejects), and never rejects a consensus-valid job on local policy. Prechecks (duplicate-wtxid,bad-missing-tx,bad-cb-decode) run before any node call. Pending proposals awaiting a provide are held per session (8, 30 s), oldest evicted; late or unknown provides getunknown-request-id, a reused idduplicate-request-id. Validation runs off the session loop (4 in flight, FIFO beyond), so aSubmitSolutionnever waits behind someone else's proposal. Nothing starts without--sv2-tp-listen; a session opts in with the flag.Why
rbitcoin has a native TP but no Core-style IPC, so today a JDS cannot use it at all: SRI's JDS validates only through Bitcoin Core's
checkBlock. This makes the TDP connection the JDS already holds sufficient. The merged proposal-check stack (#959, #961, #963, #967, #968, #969) is what makes that path fit for a JDS asking "is this template valid?" dozens of times a minute per pool: one decode per parent, memory bounded by the block, and right about spent coins after a reorg.The Core-IPC sibling of the same messages is stratum-mining/sv2-tp#137 (draft, same shape, same edge-case behaviour).
Commits
19, each Red → Green → Refactor; see
docs/sv2-template-provider.mdPlan D (D1–D12) for the per-step contracts. In order: messages and the setup flag; per-message client frame caps; theProposeTemplatehandler; the missing-transactions round trip; ordered rejection of untrusted input; a pinnedSubmitSolutionagainst a validated job; docs; the rename and field realignment to the RFC (ProposeTemplate, prefix/suffix,prev_hashin.Success);feesdropped then restored (mandatory, TP-computed); validation off the session loop; the concurrency journey reshaped after #968 made the check fast; the request/provide leg.Tests
cargo test -p rbitcoin-sv2 --lib: 31 pass. Journeys on the shared regtest pad cover: happy path with the fee total andprev_hash; missing → provide → success →SubmitSolutionadvances the tip; rejections in order (duplicate-wtxid,bad-missing-tx,bad-cb-decode, old-heightbad-cb-height, overpaying coinbasebad-cb-amount);unknown-request-id,duplicate-request-id, provide expiry, short provide; a validation never delaysRequestTransactionDataon the same session (400-input proposal, 10/10 × 3). Workspace clippy-D warnings, fmt, ast-grep, deny clean locally.Docs
docs/sv2-job-validation.mdis the proposed wire contract (our draft for #239; 0x78/0x7b numbers are ours, plebhash only said "after 0x76"). Plan D indocs/sv2-template-provider.md, operator note indocs/operator/interfaces.md, COMPAT row,changelog.d/sv2-job-validation.md.Still open in #239 (will track)
Names/numbers of the provide pair; whether the provide leg gets its own
.Error; behaviour at a TP's in-flight bound (this queues, sv2-tp refuses); retention cap value; whetherexcess_datastays.🤖 Generated with Claude Code