Skip to content

fix(pipeline): a clamped pack's hidden rows leave no trace in its counts - #9980

Open
olety wants to merge 12 commits into
mainfrom
opus-unbuilt3-c/d19-gate-after-clamp
Open

olety wants to merge 12 commits into
mainfrom
opus-unbuilt3-c/d19-gate-after-clamp

Conversation

@olety

@olety olety commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

What

[D19-GATE-COUNTS-CLAIMS-THE-DISCLOSURE-CLAMP-REFUSES] (P2): in a context pack clamped for an audience, a dead claim the audience may not read (for example a Proposed claim marked Tier A) still counted in PackStats::claims_suppressed and made an empty pack read AllActivated in place of NoData. The audience learned one bit: a claim it may not read matched. The phonetic code rung's half landed with #9911; this PR closes the rest.

A row the audience could not read whatever its status (DisclosureContext::admits_row_ignoring_status, the door #9911 added) is now kept from every check that records a refusal in the run:

  1. Shared fetch admission (pipeline/filters.rs, pipeline_candidate_matches_filters_and_gate): the clamp's own-row test now runs where a refusal would be recorded: at the entity-type authority's refusal and just before the D19 gate. Before, both recorded first and the clamp asked last. The checks that record nothing (types, times, codebase, criticality) still run first. This covers every bounded vector/text/temporal fetch, the media lists and the PPR walk's visibility. The predicate's verdict does not change: the full clamp test with the decoded body still runs after the gate.

  2. Text gate-widening probe (execution/channels/text.rs): such a posting is no posting of the run, the same as a device-local one. FIX 199 imports every probe rejection into the run's gate, so before, a hidden dead claim counted as suppressed through the probe.

  3. Lists no bounded fetch asked (scoped_channels.rs retain_rows_the_clamp_admits, called from the run's per-list step in execution/channels/mod.rs): an unnarrowed text, HyDE-retry or temporal list held every row it ranked. A hidden row took a fusion rank and counted as a candidate in scope, and the run's D19 gate (inside the first blend and over the fused union) recorded its refusal. Such a row now leaves the list before the blend, as the phonetic code rung's exact hits do since pipeline: ask the call's scope of the phonetic exact hits before the blend #9911. A hidden row that meets every other check of the call is judged in caches of its own and still counts among the pack's clamped rows (disclosure_clamped). The phonetic step and this one share that judgement (meets_every_check_but_the_clamp). (A pack's vector walk is narrowed to the memory lane, in caches of its own whose rejections never join the run, so it never counted a hidden claim.)

  4. Fetch windows (REVIEW-9980 round 1): hidden rows that outrank a readable row no longer take its place in a fetch, so the readable row reaches the run.

    • Text: the memory lane's category read (text.rs search_text_candidates, for the primary read and the HyDE retry) asks the clamp's own-row test after the lane, so a hidden row takes no place in the BM25 window.
    • Temporal: the clamp joins the channel's reader beside the scope and the made_by slice (scoped_channels.rs scoped_temporal_results, the ReadableRows reader of fix(pipeline): a temporal scan cap asks the reader before it drops a row [TEMPORAL-READER-SCAN-CAP-REFUSED-ROWS] #9968). A hidden row counts toward no stop and takes no place under a scan's cap, in the bound or in the cut. A clamped pack's unnarrowed list now goes through that reader, keeps the cut's readable rows, and is cut to its limit as before.
    • Both hand back the hidden rows that ranked inside the first limit places, the places a read without the clamp gave them (ChannelAccumulator::clamped_at_fetch). Those that meet every other check count among the clamped rows (count_rows_clamped_at_fetch), so clamped_out keeps the count it had when they reached the blend.

clamp_hides_row (filters.rs) is the one test that sites 2, 3 and 4 share.

Behaviour

  • Pack with no clamp: unchanged. Each new branch runs only when filters.disclosure is set, and admits_row_ignoring_status admits everything when the context does not clamp.
  • Clamped pack:
    • A hidden row takes no place in the memory lane's text window or in the temporal channel's window and cut, so a readable row it outranked now reaches the run: the pack reads as it does without the hidden rows.
    • A hidden row takes no rank in an unnarrowed list, as at a bounded fetch. Removing it changes that list's score scale in the fusion, so readable rows can change order, and the result cut can keep a different readable set. No hidden row is served either way.
    • A hidden row of an unnarrowed list that passes every other check counts in clamped_out whatever its score. This is the rule the phonetic rung has followed since pipeline: ask the call's scope of the phonetic exact hits before the blend #9911.
    • The count is taken before the turn fold, the rerank and RET-01 abstention. So a hidden MESSAGE and its hidden TURN held by two lists count as two rows, and a hidden row that abstention would have emptied out now counts. It is the owner-side count only; the pack the audience receives does not carry it.
    • Where clamped_out can now differ from before. This is the owner-side count only: no audience-visible output reads it, and clamped_out_under filters it.
      • With a port walk pending, the text read judges postings in index order, so it counts the first limit hidden lane rows it meets rather than the best ranked.
      • With the turn fold on, the count is taken against the last, longer read rather than against the fold's cut.
      • The temporal reader's window can grow past the one a call without the clamp held, because hidden rows no longer stop its rounds. A scan over its cap drops hidden rows uncounted.
      • In a strict- or corpus-scoped pack, a hidden row inside the temporal cut that meets every other check now counts. Before, the scope cut dropped it uncounted.
    • Not changed here: a live hidden row that a bounded fetch drops (a claim search, or a pack's vector walk) counts in no clamped_out, on main as on this head. Filed as [CLAMPED-COUNT-MISSES-BOUNDED-FETCH-DROPS].
    • Not changed here: a dead claim the clamp refuses only for the sources of its words (the row itself readable) still reaches the gate and counts, as in the phonetic rung. The own-row test cannot see word sources, and the full test needs a live row. Filed as [CLAMP-WORD-SOURCE-HIDDEN-DEAD-CLAIM-COUNTS].
    • A hidden row of a kind the entity-type authority refuses is still recorded in read_suppressed by the per-list type step, on main as on this head. That set feeds scoped-read receipts only, not the pack.
  • Cost, only under a clamp:
    • one own-row disclosure test for each fetched candidate that passes the checks that record nothing (it reaches the gate) or that the type authority refuses;
    • one more for each row of each ranked list before the blend;
    • one for each lane row the memory lane's text read judges until its window holds limit readable rows, each a deadline checkpoint as at the scope's admission;
    • one, memoized, for each row the temporal window holds: the rounds' stops and the cut ask the reader, and the list walks the whole held window once.
      This is an estimate by reading, not a measurement.

Stored change: none. No verb, no ABI.

Tests

crates/oneiron/tests/clamp_before_claim_gate.rs, run with RUSTFLAGS="--cfg oneiron_test_support" cargo test --locked -p oneiron --all-features --test clamp_before_claim_gate:

  • Red on main: a clamped text search reads alike with and without a hidden proposed claim (Tier A), and counts no suppressed claim. Two tests, each FAILBEFORE D19-CLAMP: a claim search (the bounded fetch, change 1) and an unnarrowed search (the widening probe and the fused union, changes 2 and 3 together).
  • The fixture checks that the clamp refuses the hidden claim even served as a live revision: Tier A hides it, not its status.
  • Red on the pre-fix head (REVIEW-9980 round 1), each read with and without hidden proposed claims that outrank a readable proposed claim:
    • a_clamped_text_search_reads_alike_with_hidden_claims_that_outrank_a_readable_one (FAILBEFORE REVIEW-9980-TEXT-WINDOW): 100 hidden claims matching both words of cobalt zircon, one readable claim matching one word in a field as long;
    • a_clamped_temporal_search_reads_alike_with_hidden_claims_nearer_its_anchor (FAILBEFORE REVIEW-9980-TEMPORAL-CUT): Occurred mode, 100 hidden claims at the anchor's midpoint, the readable claim half a sigma off it.
  • Guards, green on main and on the head:
    • a proposed claim the audience may read still counts as suppressed, with the owner's empty reason, on both paths;
    • a live hidden claim reached by an unnarrowed search still counts among the clamped rows;
    • a live hidden claim inside an unnarrowed temporal window still counts among the clamped rows.

Red pins: branch opus-unbuilt3-c/d19-gate-after-clamp-red, which is main plus the round-0 test file (the two D19-CLAMP tests), and commit 7efd5e4f559e, the tests on the pre-fix head (the two REVIEW-9980 tests).

Reviews

  • Sol (xhigh) and a blind Opus second review, both read-only:
    • Neither found a code defect. Neither found a path left by which a row hidden by its own row test still puts a refusal in the run.
    • Both found the predicate's verdict unchanged for every row: a row the own-row test refuses also fails the full clamp test.
  • Their test findings, all fixed in the test rework:
    • A pack's vector walk never counted a hidden claim, so the vector tests stopped at their fixture.
    • The readable-Proposed guard's fixture asked the full clamp, which needs a live row.
  • Opus P3 (cost) fixed: the early test now runs only where a refusal is recorded.
  • The other P3s (count drift; word sources; the bounded-fetch clamped count; read_suppressed) are stated above.
  • Pipeline review round 1 (REVIEW-9980, two P1s): hidden rows that outranked a readable row filled the memory lane's text window and the unnarrowed temporal cut. Both are fixed by change 4 (tests 7efd5e4f559e, fix e5e453e29b31).
  • Reviews of that fix (the Sol slots were full, so the first review ran on Spark max; then a blind Opus review):
    • Both found that the first version's temporal count was always empty, because the window cut asks the reader about the row itself. Fixed in 8b8365489679 with WindowCut::ranks_inside_bound; the new temporal guard covers it.
    • Opus at 8b8365489679: no P1 or P2, and both scenarios are fixed. Its P3s:
      • a deadline checkpoint for the clamped text read: fixed in 5abe5e2135fd;
      • count drift: stated above;
      • temporal cost: stated above.
    • Spark P2, that a narrowed temporal list lost its count: not taken. On main and before this fix, the scope cut (truncate_to_scope) dropped hidden rows from a narrowed list before any count.
    • Already the case on main, not in this PR's scenarios: BM25 df counts hidden postings, so hidden rows can still reorder readable text rows through IDF. Filed as [CLAMPED-BM25-IDF-COUNTS-HIDDEN-POSTINGS].

…ead (red)

[D19-GATE-COUNTS-CLAIMS-THE-DISCLOSURE-CLAMP-REFUSES]: a proposed claim the
pack's audience may not read (Tier A) counts in claims_suppressed and makes
an empty clamped pack read AllActivated, for a bounded text or vector fetch
(a claim search) and for a list that reaches the blend unasked. Guards: a
proposed claim the audience may read still counts, and a live hidden claim
still counts among the pack's clamped rows.
[D19-GATE-COUNTS-CLAIMS-THE-DISCLOSURE-CLAMP-REFUSES]: in a pack clamped for
an audience, a row the audience could not read whatever its status
(DisclosureContext::admits_row_ignoring_status) is now kept from every check
that records a refusal in the run, so a dead claim it may not read counts in
no claims_suppressed and leaves the empty reason as it is:

- the shared fetch admission asks the clamp's own-row test before the
  entity-type authority and the D19 gate record anything;
- the text channel's gate-widening probe treats such a posting as no posting
  of the run, as it treats a device-local one (its rejections join the gate);
- a ranked list no bounded fetch asked (an unnarrowed vector, text or
  temporal list) drops such a row before the blend, as the phonetic code
  rung does since #9911, so it takes no rank and counts as no candidate in
  scope; one that meets every other check, judged in caches of its own,
  still counts among the pack's clamped rows.

A pack with no clamp is unchanged.
…in-scope count

An empty clamped pack reports the rows it surfaced as its candidates in
scope, never the pre-clamp count the owner's pack reports, so the guard for
a proposed claim the audience may read compares the empty reason and the
suppressed claims only.
@coderabbitai

coderabbitai Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

  • Run on-demand review

This review includes 17 billable files and costs up to $4.25.

  • Ask an admin to make reviews automatic

Open in CodeRabbit

Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing.

Or wait 57 minutes for your next included review.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available. Your 151 included PR review attempts over the past 7 days set your current allowance at 1 review per hour.

Learn how review limits work.

Review configuration:

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 4317485 and 5e856ef.

Files selected for processing: 17

  • crates/oneiron/src/bm25/mod.rs
  • crates/oneiron/src/bm25/query.rs
  • crates/oneiron/src/bm25/scoped.rs
  • crates/oneiron/src/bm25/scoring.rs
  • crates/oneiron/src/bm25/tests.rs
  • crates/oneiron/src/mail_pack/compliance.rs
  • crates/oneiron/src/memory/session_reads.rs
  • crates/oneiron/src/pipeline/execution/channels/admit.rs
  • crates/oneiron/src/pipeline/execution/channels/mod.rs
  • crates/oneiron/src/pipeline/execution/channels/text.rs
  • crates/oneiron/src/pipeline/filters.rs
  • crates/oneiron/src/pipeline/scoped_channels.rs
  • crates/oneiron/src/pipeline/temporal_bound.rs
  • crates/oneiron/src/ports/lmdb_retrieval.rs
  • crates/oneiron/src/ports/retrieval.rs
  • crates/oneiron/src/vault/search_retrieval.rs
  • crates/oneiron/tests/clamp_before_claim_gate.rs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

…gate reaches

A context pack runs in the memory lane, which narrows its vector walk in
caches of its own whose rejections never join the run, so a pack's vector
search never counted a hidden claim and its tests stopped at the owner's
fixture. The text channel reaches the gate both ways: a claim search's
bounded fetch, and an unnarrowed search's widening probe and fused union.
The fixture asks the clamp of a hidden claim served as a live revision (Tier
A, not its status, hides it), and the guards search text only.
…l records

Review of #9980: the clamp's own-row test ran before the cheap checks that
record nothing (types, times, codebase, criticality), so a clamped scan paid
it for every posting. It now runs only where a refusal would be recorded: at
the entity-type authority's refusal and just before the D19 gate. The verdict
is unchanged. The per-list step's doc names the lists it meets: a pack's
vector walk is narrowed to its memory lane.
@olety
olety marked this pull request as ready for review November 20, 2026 19:32
@greptile-apps

greptile-apps Bot commented Nov 20, 2026

Copy link
Copy Markdown

Automatic reviews are disabled for this repository. Configure in Code Review Settings.

…window or temporal cut

REVIEW-9980 round 1: proposed claims the audience may not read that outrank
a readable proposed claim filled the unnarrowed text scan's window and the
temporal channel's cut, and left the lists only before the blend, so the
clamped pack read NoData with no suppression where it read AllActivated
with one. A guard keeps a live hidden claim of an unnarrowed temporal list
among the pack's clamped rows.
…den rows no place

REVIEW-9980 round 1. The clamp's own-row test ran only before the blend, so
rows the audience could not read whatever their status still took places
in the unnarrowed fetches: hidden claims that outranked a readable one
filled the memory lane's text window (category BM25) or the temporal
channel's window, scan caps and cut, and the readable row never reached
the run. The pack then read NoData with no suppression where it read
AllActivated.

- Text: the memory lane's category read asks the clamp after the lane, so a
  hidden row takes no place in its window.
- Temporal: the clamp joins the channel's reader (the scope, then the
  made_by slice), so a hidden row counts toward no stop and takes no place
  under a scan's cap or in the bound; a clamped pack's unnarrowed list is
  then cut as before.
- Both hand back the hidden rows that ranked inside the places the read or
  cut gave them before; the pack counts those that meet every other check
  among the rows its clamp withheld, as it did when they reached the blend.
…lace in the bound

The window cut asks the reader of the row itself, so it never keeps a
hidden row, and the count of hidden rows among the cut's rows was always
empty: a live hidden claim of an unnarrowed temporal list no longer counted
among the pack's clamped rows. The count now walks the rows the bound holds
with no reader asked (WindowCut::ranks_inside_bound), best first, in the
lane, and takes the hidden ones among the first `limit`.
…ch judged row

Review of the round-1 fix (Opus P3): a read that judges past many hidden
rows ran with no deadline checkpoint, unlike the scope's admission, which
checks in at every candidate. Under a clamp each judged lane row now
checks in. Docs say what the owner-side clamped count takes on the
pending-walk text read and in a temporal window that grows past the one
a call without the clamp held.
…l cut counts them

REVIEW-9980 round 2. In a clamped pack, hidden proposed claims holding a
readable claim's word counted in that word's document frequency, so live
claims outranked the readable one and filled the text window: the pack
counted no suppressed claim. A made_by slice narrows the temporal list,
and its cut handed back no hidden row, so a live hidden claim inside the
window counted among no clamped rows.

The text helpers now blend no recency, as both reviews' scenarios read:
with the blend, the text window holds 320 rows, not 80.
…c, and a narrowed temporal cut hands them back

REVIEW-9980 round 2.

- BM25: under a clamp, a term's document frequency counts only the
  postings of rows the audience could read (a live hint's claim, or the
  row itself; a dead hint counts as hidden). The pipeline's text stage
  and HyDE retry pass the clamp's own-row test through
  Bm25SearchOptions::hidden_row and TextQuery::hidden_row. Hidden
  postings still score, for the owner-side count of the rows the clamp
  withheld; admission keeps them out. The document count and the fields'
  average lengths stay the vault's.
- Temporal: a narrowed list (a scope or a made_by slice) under a clamp
  hands back the hidden rows that the scope and slice admit inside the
  bound a reader without the clamp gives (WindowCut::rows_inside_bound_where),
  among the first limit lane rows, as the unnarrowed list does.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant