Skip to content

feat(95): surface table distrust via tables_trust.json + doc-level pointer - #100

Merged
r-uben merged 2 commits into
mainfrom
feat/95-tables-trust-sidecar
Jul 30, 2026
Merged

feat(95): surface table distrust via tables_trust.json + doc-level pointer#100
r-uben merged 2 commits into
mainfrom
feat/95-tables-trust-sidecar

Conversation

@r-uben

@r-uben r-uben commented Jul 30, 2026

Copy link
Copy Markdown
Owner

Summary

Makes table distrust visible to consumers via a tables_trust.json sidecar plus a
document-level pointer.

Dual-pass already detects untrusted tables and records them in audit_log.json,
but on the OBR EFO November 2022 reference run the assembled markdown looked like
clean prose and tables while 20 of the 20 pages carrying tables were flagged,
and metadata.json said only:

status: partial
error:  "page(s) 46 produced no usable output"

One page named. Flag-only is a defensible write policy; it is a bad read policy
when the durable artifacts hide the flags.

Scope / non-goals

  • No write-policy change. --auto-patch-tables stays opt-in. This is a
    read-only derivation over existing audit events.
  • The assembled .md is untouched. Surfaced table distrust: dualpass flags never reach the consumer .md #95 offers "callout and/or sidecar";
    the sidecar is chosen because the markdown is produced by
    _rewrite_all_fragments, whose byte-identity with whole-doc assembly is guarded
    by golden tests. Prose-only pages therefore stay unmarked.

Implementation

socr/core/tables_trust.py derives tables_trust.json beside audit_log.json.
Top-level summary a consumer can gate on without walking per-page records or
parsing the audit log
: untrusted_page_count, untrusted_pages,
table_flags_n, counts_by_kind, patch_eligible_pages.

Distrust taxonomy is explicit, with exclusions documented in the module docstring:
resolved disagreements (dualpass_patched, table_header_repair), passes
(native_table_verifier_exact_pass), and page_failed (not table-specific, and
already surfaced via LOST_CONTENT_NOTE; failed table pages still carry
table_region_unverifiable).

PATCH_ELIGIBLE_NOTE is named in tables/reconcile.py so patch-eligibility is
identified by constant comparison instead of parsing prose out of the audit log.

Doc-level signal: a constraint worth flagging to the reviewer

ocr_output_contract.DocMetadata has a fixed field setstatus, checksum, model, backend, processing_time, timestamp, output_path, pages, error, fingerprint, mode, fell_back_from_whole_pdf. There is no slot for a trust count. So the signal
rides in error, prefixed with a stable token for substring gating, appended so a
lost-content or halt reason still reads first.

It reports counts, not the page list: enumerating 20 page numbers made the
field grow with document size. tables_trust.json carries the list.

EngineResult.success derives only from status, so writing error on an
otherwise-clean run is inert for control flow — but that is worth a reviewer's
second look.

What this does NOT establish

Validating the derivation against the real audit log broke a claim in the
bake-off note (corrected there):

  • Page 66 is flagged, by value_guard_row_count_warning. So 20 of 20
    table pages are flagged and there are no negative controls.
  • Recall is trivially 100%; precision cannot be measured and is visibly poor
    page 45 scores 100.0% cell exactness and is flagged, page 67 94.1%, page 66
    91.2%.

So this sidecar is a triage list ("look at these"), not a trust verdict
("these are wrong"). It satisfies #95's acceptance — an outsider can name an
untrusted page without reading audit_log.json by hand — but a consumer must not
read untrusted_pages as the broken set. Severity ranking is a follow-up.

Test plan — observed results

  • pytest tests/test_gh95_tables_trust.py -q17 passed
  • pytest tests/ -q1252 passed, 1 xfailed
  • ruff format --check + ruff check on all four touched files → clean
  • Validated against the real audit log: reproduces 20 untrusted pages, 46 flags,
    counts_by_kind matching the run, and 8 patch-eligible pages.

All tests are pure derivations over AuditEvent objects — no engine, no provider
ladder, so hermetic in CI without patching _available_engines_for_agentic.

Includes a taxonomy guard that fails if a watched kind is never emitted anywhere
in src/, to stop a typo silently disabling a signal.

Merge order

table_row_repetition_truncated is watched here but emitted by #97 (PR #99). The
taxonomy test lists it as pending and is forward-compatible in both directions —
deliberately not a tripwire that fails once #99 lands. Either merge order is
safe.

Closes #95

r-uben added 2 commits July 30, 2026 16:47
…inter

Dual-pass already detects untrusted tables and records them in audit_log.json,
but the assembled markdown looked like clean prose and tables while 20 of the 20
table pages of the OBR EFO November 2022 reference run carried table flags, and
metadata.json named exactly one page ("page(s) 46 produced no usable output").
Flag-only is a defensible write policy; it is a bad read policy when the durable
artifacts hide the flags.

Adds socr/core/tables_trust.py: a read-only derivation over the audit events
producing tables_trust.json beside audit_log.json, with a top-level summary
(untrusted_page_count, untrusted_pages, table_flags_n, counts_by_kind,
patch_eligible_pages) a consumer can gate on without walking the per-page
records or parsing the audit log.

Sidecar rather than an inline callout in the .md: the assembled markdown is
produced by _rewrite_all_fragments, whose byte-identity with whole-doc assembly
is guarded by golden tests, and #95 accepts "and/or" a sidecar. Prose-only pages
therefore stay unmarked and no write policy changes - --auto-patch-tables remains
opt-in.

Doc-level signal rides in metadata.json's error field: ocr-output-contract's
DocMetadata has a fixed field set with no slot for a trust count. Prefixed with a
stable token for substring gating, appended so a lost-content or halt reason
still reads first, and reporting counts rather than the page list so the field
cannot grow with document size (20 of 20 pages flagged on the reference run).

Names PATCH_ELIGIBLE_NOTE in tables/reconcile.py so patch-eligibility is
identified by constant comparison instead of parsing prose out of the audit log.

Excluded from the distrust taxonomy, with reasons in the module docstring:
resolved disagreements (dualpass_patched, table_header_repair), passes
(native_table_verifier_exact_pass), and page_failed (not table-specific and
already surfaced via LOST_CONTENT_NOTE; failed table pages still carry
table_region_unverifiable).

Tests: 17 new, all pure derivations over AuditEvent objects - no engine, no
provider ladder, so hermetic in CI without patching
_available_engines_for_agentic. Includes a taxonomy guard that fails if a watched
kind is never emitted anywhere in src/ (it caught the GH-97 kind not yet
being emitted on this branch). Validated against the real 19-page audit log. Full suite
1252 passed, 1 xfailed. ruff clean.

Closes #95
A live 4-page run wrote this into metadata.json:

  untrusted tables on page(s) 4 page(s), 11 flag(s) (see tables_trust.json)

TRUST_NOTE_PREFIX still ended in "on page(s)" from the earlier version that
enumerated page numbers; switching the note to counts left the stale fragment in
place. Prefix is now "untrusted tables" and the note composes as

  untrusted tables on 4 page(s), 11 flag(s) (see tables_trust.json)

The prefix stays a stable substring-gating token, so no consumer contract changes.

Every existing test asserted on substrings ("2 page(s)" in note), which is why
none caught it. Adds a whole-string assertion.
@r-uben
r-uben force-pushed the feat/95-tables-trust-sidecar branch from 397b278 to a11e0d4 Compare July 30, 2026 14:49
@r-uben
r-uben merged commit acfc23f into main Jul 30, 2026
2 checks passed
@r-uben
r-uben deleted the feat/95-tables-trust-sidecar branch July 30, 2026 14:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Surfaced table distrust: dualpass flags never reach the consumer .md

1 participant