Skip to content

Classify verify-raw diagnostics on whole phrases, after stripping colour codes - #336

Merged
MaxGhenis merged 2 commits into
mainfrom
nzhub/verify-raw-diagnostics
Oct 11, 2026
Merged

MaxGhenis merged 2 commits into
mainfrom
nzhub/verify-raw-diagnostics

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Follow-up to #334, fixing the diagnostic nit from its review.

The problem

chronicle verify-raw decides between missing_remote and fetch_error by looking for marker text in Wrangler's stderr. The review of #334 found two misclassifications. Both only affect entries that already fail, so neither could ever report verified:

  • Colour codes inside the missing-object phrase made it a fetch_error.
  • Unrelated text that merely mentioned NoSuchKey made it missing_remote.

The fix

_is_missing_object_diagnostic in chronicle/artifacts.py now:

  • strips ANSI colour sequences and collapses whitespace before matching;
  • matches a marker only as a whole phrase, so NoSuchKeyHandler isn't NoSuchKey;
  • counts a marker only on a line that holds nothing but markers (such as NoSuchKey: specified key does not exist) or is an error line (it contains error, failed or Wrangler's ✘).

Known limit, pinned in a test: an error line that merely mentions a marker, such as error: could not reach bucket; NoSuchKey handling disabled, still gives missing_remote. Both statuses are failures, so this misnames a failure but can never pass an object.

Nothing else in verify-raw changes.

Invariants and how they're checked

  • Colour codes anywhere in the message still give missing_remote. One test inserts a colour sequence at every position of [ERROR] The specified key does not exist.
  • A marker inside a longer identifier, or on a line that reports no error, gives fetch_error. Five cases cover this. An error line that mentions a marker is the known limit above.
  • Existing behaviour holds. The bare markers, the S3-style Code: message line and Wrangler's real captured stderr all still give missing_remote.

Tests

  • tests/test_chronicle_verify_raw.py: 37 passed (25 existing, 12 new).
  • tests/test_chronicle_artifacts.py: 36 passed.
  • Three in-memory mutants each fail the new tests: the old substring logic (8 failures), no colour stripping (2) and never-missing (7).
  • Live, with the real Wrangler and this branch's code (hub-reported; no log is kept in the repo): an object absent at its canonical key reports missing_remote, and the published IRD wage and salary workbook reports verified.
  • Ruff 0.14.10 check and format pass.

Repository tooling only: no source package, fact or bundle pin changes.

Review

An independent GPT-6.1 Sol review at a1ba3a3d returned APPROVE_WITH_NITS, with no blocking defect:

  • both original probes pass;
  • 36 + 36 tests pass;
  • all 11 new cases fail under mutants (five mutants: 8/2/7/2/3 failures).

Its nits, answered in fc9532ad:

  • the changelog's incidental-marker claim is now qualified, and the error-line limit is pinned as a test case;
  • the captured Wrangler bytes are labelled as the hub's observation.

It also noted, without asking for changes: constructed JSON or multi-line messages give fetch_error, and OSC or 8-bit CSI sequences aren't stripped. None of these can produce verified; widening the matching waits for a real captured case.

It marked "Needs Max: yes" because a changelog fragment is public. The hub doesn't hold on that: a developer-tool changelog line is not a public claim in the sense of Max's merge rule.

🤖 Generated with Claude Code

MaxGhenis and others added 2 commits October 10, 2026 20:07
…ripping colour codes

The review of #334 found two misclassifications among failing entries. Colour
codes inside the missing-object phrase produced fetch_error, and unrelated text
that merely mentioned NoSuchKey produced missing_remote. Neither could report
verified.

_is_missing_object_diagnostic now strips ANSI sequences and collapses
whitespace, matches markers only as whole phrases (not inside a longer
identifier), and counts a marker only on a line that holds nothing but markers
or is an error line.

Invariants, each with tests:
- Colour codes at any position of the missing-object message still give
  missing_remote (exhaustive over every insertion point).
- A marker inside a longer identifier, or on a line reporting no error, gives
  fetch_error.
- The bare markers and Wrangler's real captured output keep giving
  missing_remote; no status other than the missing/fetch-error split changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review r1 of #336: an error line that merely mentions NoSuchKey still gives
missing_remote, so the fragment overstated the fix. Say so plainly, pin that
case in the parametrised test, and label the captured Wrangler bytes as the
hub's observation rather than a retained log.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis merged commit e3a09e5 into main Oct 11, 2026
3 checks passed
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Merged by the NZ hub (session local_e407a527) at fc9532a with --merge. Gates: gh pr checks exit 0; MERGEABLE; not draft; no CHANGES_REQUESTED review. Reviewed head: fc9532a. Independent GPT-6.1 Sol reviews: round 1 APPROVE_WITH_NITS at a1ba3a3 (36 + 36 tests; 11 new cases each fail under mutants), round 2 APPROVE_WITH_NITS at fc9532a, Needs Max: no (37 tests; changelog now states the error-line limit, which is pinned as a test). Round 1 marked a developer-tool changelog line as needing Max; the hub does not hold on that. The remaining docstring nit (artifacts.py _is_missing_object_diagnostic still says incidental mentions stay a fetch error) goes into the discovery follow-up PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant