feat(95): surface table distrust via tables_trust.json + doc-level pointer - #100
Merged
Conversation
This was referenced Jul 30, 2026
Merged
r-uben
force-pushed
the
feat/95-tables-trust-sidecar
branch
from
July 30, 2026 14:21
0601aa0 to
397b278
Compare
…inter
Dual-pass already detects untrusted tables and records them in audit_log.json,
but the assembled markdown looked like clean prose and tables while 20 of the 20
table pages of the OBR EFO November 2022 reference run carried table flags, and
metadata.json named exactly one page ("page(s) 46 produced no usable output").
Flag-only is a defensible write policy; it is a bad read policy when the durable
artifacts hide the flags.
Adds socr/core/tables_trust.py: a read-only derivation over the audit events
producing tables_trust.json beside audit_log.json, with a top-level summary
(untrusted_page_count, untrusted_pages, table_flags_n, counts_by_kind,
patch_eligible_pages) a consumer can gate on without walking the per-page
records or parsing the audit log.
Sidecar rather than an inline callout in the .md: the assembled markdown is
produced by _rewrite_all_fragments, whose byte-identity with whole-doc assembly
is guarded by golden tests, and #95 accepts "and/or" a sidecar. Prose-only pages
therefore stay unmarked and no write policy changes - --auto-patch-tables remains
opt-in.
Doc-level signal rides in metadata.json's error field: ocr-output-contract's
DocMetadata has a fixed field set with no slot for a trust count. Prefixed with a
stable token for substring gating, appended so a lost-content or halt reason
still reads first, and reporting counts rather than the page list so the field
cannot grow with document size (20 of 20 pages flagged on the reference run).
Names PATCH_ELIGIBLE_NOTE in tables/reconcile.py so patch-eligibility is
identified by constant comparison instead of parsing prose out of the audit log.
Excluded from the distrust taxonomy, with reasons in the module docstring:
resolved disagreements (dualpass_patched, table_header_repair), passes
(native_table_verifier_exact_pass), and page_failed (not table-specific and
already surfaced via LOST_CONTENT_NOTE; failed table pages still carry
table_region_unverifiable).
Tests: 17 new, all pure derivations over AuditEvent objects - no engine, no
provider ladder, so hermetic in CI without patching
_available_engines_for_agentic. Includes a taxonomy guard that fails if a watched
kind is never emitted anywhere in src/ (it caught the GH-97 kind not yet
being emitted on this branch). Validated against the real 19-page audit log. Full suite
1252 passed, 1 xfailed. ruff clean.
Closes #95
A live 4-page run wrote this into metadata.json:
untrusted tables on page(s) 4 page(s), 11 flag(s) (see tables_trust.json)
TRUST_NOTE_PREFIX still ended in "on page(s)" from the earlier version that
enumerated page numbers; switching the note to counts left the stale fragment in
place. Prefix is now "untrusted tables" and the note composes as
untrusted tables on 4 page(s), 11 flag(s) (see tables_trust.json)
The prefix stays a stable substring-gating token, so no consumer contract changes.
Every existing test asserted on substrings ("2 page(s)" in note), which is why
none caught it. Adds a whole-string assertion.
r-uben
force-pushed
the
feat/95-tables-trust-sidecar
branch
from
July 30, 2026 14:49
397b278 to
a11e0d4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes table distrust visible to consumers via a
tables_trust.jsonsidecar plus adocument-level pointer.
Dual-pass already detects untrusted tables and records them in
audit_log.json,but on the OBR EFO November 2022 reference run the assembled markdown looked like
clean prose and tables while 20 of the 20 pages carrying tables were flagged,
and
metadata.jsonsaid only:One page named. Flag-only is a defensible write policy; it is a bad read policy
when the durable artifacts hide the flags.
Scope / non-goals
--auto-patch-tablesstays opt-in. This is aread-only derivation over existing audit events.
.mdis untouched. Surfaced table distrust: dualpass flags never reach the consumer .md #95 offers "callout and/or sidecar";the sidecar is chosen because the markdown is produced by
_rewrite_all_fragments, whose byte-identity with whole-doc assembly is guardedby golden tests. Prose-only pages therefore stay unmarked.
Implementation
socr/core/tables_trust.pyderivestables_trust.jsonbesideaudit_log.json.Top-level summary a consumer can gate on without walking per-page records or
parsing the audit log:
untrusted_page_count,untrusted_pages,table_flags_n,counts_by_kind,patch_eligible_pages.Distrust taxonomy is explicit, with exclusions documented in the module docstring:
resolved disagreements (
dualpass_patched,table_header_repair), passes(
native_table_verifier_exact_pass), andpage_failed(not table-specific, andalready surfaced via
LOST_CONTENT_NOTE; failed table pages still carrytable_region_unverifiable).PATCH_ELIGIBLE_NOTEis named intables/reconcile.pyso patch-eligibility isidentified by constant comparison instead of parsing prose out of the audit log.
Doc-level signal: a constraint worth flagging to the reviewer
ocr_output_contract.DocMetadatahas a fixed field set —status, checksum, model, backend, processing_time, timestamp, output_path, pages, error, fingerprint, mode, fell_back_from_whole_pdf. There is no slot for a trust count. So the signalrides in
error, prefixed with a stable token for substring gating, appended so alost-content or halt reason still reads first.
It reports counts, not the page list: enumerating 20 page numbers made the
field grow with document size.
tables_trust.jsoncarries the list.EngineResult.successderives only fromstatus, so writingerroron anotherwise-clean run is inert for control flow — but that is worth a reviewer's
second look.
What this does NOT establish
Validating the derivation against the real audit log broke a claim in the
bake-off note (corrected there):
value_guard_row_count_warning. So 20 of 20table pages are flagged and there are no negative controls.
page 45 scores 100.0% cell exactness and is flagged, page 67 94.1%, page 66
91.2%.
So this sidecar is a triage list ("look at these"), not a trust verdict
("these are wrong"). It satisfies #95's acceptance — an outsider can name an
untrusted page without reading
audit_log.jsonby hand — but a consumer must notread
untrusted_pagesas the broken set. Severity ranking is a follow-up.Test plan — observed results
pytest tests/test_gh95_tables_trust.py -q→ 17 passedpytest tests/ -q→ 1252 passed, 1 xfailedruff format --check+ruff checkon all four touched files → cleancounts_by_kindmatching the run, and 8 patch-eligible pages.All tests are pure derivations over
AuditEventobjects — no engine, no providerladder, so hermetic in CI without patching
_available_engines_for_agentic.Includes a taxonomy guard that fails if a watched kind is never emitted anywhere
in
src/, to stop a typo silently disabling a signal.Merge order
table_row_repetition_truncatedis watched here but emitted by #97 (PR #99). Thetaxonomy test lists it as
pendingand is forward-compatible in both directions —deliberately not a tripwire that fails once #99 lands. Either merge order is
safe.
Closes #95