Skip to content

fix: report receipt-less ok observations as unverifiable in receipts mode - #57

Open
Abhishek-B-R wants to merge 1 commit into
getcomputable:mainfrom
Abhishek-B-R:fix/receipts-mode-empty-receipts
Open

Abhishek-B-R wants to merge 1 commit into
getcomputable:mainfrom
Abhishek-B-R:fix/receipts-mode-empty-receipts

Conversation

@Abhishek-B-R

@Abhishek-B-R Abhishek-B-R commented Sep 24, 2026 •

Copy link
Copy Markdown

Problem

The as-published day files currently carry receipts: [] on every observation, for example:

curl -s https://data.getcomputable.com/H100/published/observations/2026/09/23.json | jq '[.data.observations[] | (.receipts | length)] | unique'
[0]

The same holds for B200 on 2026-09-01, 2026-09-10 and 2026-09-23. The versioned day files (H100/v7/observations/2026/09/23.json) and latest.json still carry receipts.

./reproduce --receipts reads the as-published files by default, rebuilds zero passing sources and reports every printed value as a mismatch. Re-checked today on main (0530437):

$ ./reproduce --receipts h100 2026-09-23T12
H100 2026-09-23T12:45 recomputed NO-PRINT published 3.522611 (band 0.545911) MISMATCH digest OK version 7 methodology_id h100_sxm_v1_calc_v16
  published status ok but only 0 observed passing disclosed sources (< min_sources_to_publish 5): no composite is recomputable
summary: 4 observation(s): 0 MATCH, 4 MISMATCH, 0 degraded
(exit 1)

Expected: no MISMATCH. The published values are fine: ./reproduce --receipts --version 7 h100 2026-09-23T12 reads the versioned file and gives 4 MATCH, exit 0 (and --version 5 b200 2026-09-10T03 the same). An empty receipts array means nothing was disclosed to recompute from, which is a could-not-verify outcome rather than a disagreement.

Fix

  • published/verify.py: an ok observation with an empty receipts array gets a new unverifiable verdict with the message "observation carries no receipts". No-print observations and every matching path are unchanged.
  • scripts/verify_published_record.py: prints UNVERIFIABLE for it, appends , N unverifiable to the summary only when N > 0 (so the summary of every passing run, and the live test regex that parses it, stay byte-identical), and exits 2 like the other could-not-verify outcomes. Without --version it also suggests rerunning with the version shown on each line.
  • README: one sentence in the --receipts paragraph.

A question for you: would you rather --receipts read the versioned receipts instead, the way #54 made --full fall back to them? That would verify these rows rather than report them as unverifiable. It is a larger change to the reader routing, so this PR keeps to the minimum, and I am happy to take it that way if you prefer. If omitting receipts from the as-published rows is not intended, the verdict change still seems right for any future receipt-less row.

Tests

  • tests/unit/test_published_verify.py: the verdict for a receipt-less ok observation.
  • tests/unit/test_published_reader_cli.py: exit code, summary and message. Both fail on main.
  • pytest: 1711 passed, 22 deselected (live). ruff check src tests scripts: clean.
  • Live, on this branch: ./reproduce --receipts h100 2026-09-23T12 reports 0 MATCH, 0 MISMATCH, 0 degraded, 4 unverifiable, exits 2, and prints the --version hint.

The commit is signed off per CONTRIBUTING.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

…mode

The as-published day files currently carry an empty receipts array on every observation, so `./reproduce --receipts` rebuilt zero passing sources and reported each printed value as a MISMATCH with exit 1. An empty receipts array is not evidence that a value is wrong; it means nothing was disclosed to recompute from.

An ok observation with no receipts now gets its own unverifiable verdict. The CLI prints UNVERIFIABLE with the reason, adds an unverifiable count to the summary only when there is one, and exits 2 like the other could-not-verify outcomes. Without --version it also points at the versioned record, whose day files keep the receipts and verify.

Validation: ruff passes and the offline suite passes; the two new tests fail on main.
Signed-off-by: Abhishek B R <byteblaster1468@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant