Skip to content

A document the tool deliberately declines reaches the command line as a traceback, not as the explanation it prepared #695

Description

@willhea

What's wrong

When DeltaTrack is handed a document it has deliberately decided it cannot compare accurately,
it explains why to a web user and crashes at a command-line user.

The engine detects one such case on purpose. Enrolled bills and public-law prints are typeset
without the printed line numbers in the left margin, and every anchor rule in the PDF parser
(the landmarks it uses to find where a section starts) depends on those numbers. Without them
the whole bill collapses into a single block and the comparison returns a confident wrong
answer. #141 (enrolled PDFs yield no anchors) is why the guard exists: rather than answer
wrongly, src/deltatrack/compare/pdf.py detects the layout and raises UnsupportedLayoutError
carrying a message written for the person who supplied the file:

This document has no printed line numbers, so DeltaTrack can't diff it accurately yet —
enrolled bills and public-law prints are typeset this way. Use the XML version of the bill
instead.

web/app.py catches it and returns that text verbatim as an HTTP 422, which is what the upload
page shows. ./diff_pdf.py catches nothing, so the same file produces a Python traceback whose
final line is that message, preceded by a stack of engine internals.

The same asymmetry applies to malformed input. Posting a file that is not XML to the endpoint
returns start_file: not XML (no leading '<') as an HTTP 415. Passing the same file to
./diff_bill.py compare raises ParseError: syntax error: line 1, column 0 uncaught.

How it surfaced

An audit of the command-line and web diff paths on develop at commit 8fb0fa16, running the
same inputs through both. The probe is docs/research/cli-ui-parity/probes/probe_flags.py on
branch worktree-audit-cli-ui-diff-parity, using an enrolled-bill fixture already committed
under tests/corpus/:

  CLI: raises UnsupportedLayoutError UNCAUGHT -> traceback for the user
  UI : HTTP 422 -> This document has no printed line numbers, so DeltaTrack can't diff it...

  CLI: uncaught ParseError: syntax error: line 1, column 0
  UI : HTTP 415 -> start_file: not XML (no leading '<').

Why it matters

The two cases are different in kind, and the distinction is the substance of this issue.

The unsupported-layout case is a designed outcome, not a failure. The tool examined the
document, reached a conclusion, and has a sentence prepared to explain it. Presenting a
designed refusal as a crash is the defect: it tells the user the tool broke when in fact the
tool worked, and it buries actionable guidance ("use the XML version") under a stack trace that
implies there is nothing to be done. An enrolled bill is a common, entirely valid file for
someone comparing appropriations bills to have on disk.

There is a second consequence for anyone scripting the tool. A traceback and a genuine crash
exit the same way, so a batch job comparing many bill pairs cannot distinguish "this document
was declined, move on" from "the tool is broken, stop". Today both are exit code 1.

The malformed-XML case is weaker. That is an unexpected input rather than a designed outcome,
and a traceback for a genuinely broken file is defensible on the command line, where the reader
is more likely to want the detail. It is included here because it is the same missing seam, not
because it carries the same argument.

This is low urgency. Nothing is wrong with the comparison itself, and a command-line user who
reads the last line of the traceback does get the right answer.

What to do

Two directions, and they can be taken separately.

  1. Present the designed refusal as a message. Catch UnsupportedLayoutError in the main()
    functions of src/deltatrack/diff_pdf.py and src/deltatrack/diff_bill.py, print
    exc.message to stderr, and exit non-zero. The user-facing wording already exists and is
    already reviewed; only its presentation on this surface is missing.

  2. Give a declined document its own exit code, distinct from an unexpected failure, so a
    caller can tell the two apart. This is the half that matters for scripted use and it is
    worth deciding deliberately, since an exit code is a small interface commitment.

Whether to extend the same treatment to parse errors is a separate call. The argument for is
consistency with the endpoint; the argument against is that a traceback is more useful than a
one-line message when the input really is broken, and command-line users are better placed to
use it. Recommendation is to do 1 and 2, and leave parse errors alone unless someone reports
being confused by one.

Verification

The fix must be shown on a real declined document rather than a synthetic exception, because
the guard's detection is itself part of the path. tests/corpus/ carries committed enrolled-bill
PDFs, and tests/test_pdf_compare.py already exercises the guard through the endpoint
(test_enrolled_upload_returns_specific_message_not_generic_422); the command-line equivalent is
the missing half.

Worth confirming the check can fail: with the handler removed, the test should see a traceback
rather than the message.

Unverified

Whether any wrapper or scheduled job currently keys on the command's exit status. Introducing a
distinct exit code for a declined document is backwards-compatible for anything treating
non-zero as failure, but not for anything matching on the value 1 specifically. No such caller
was found in this repository; callers outside it were not checked.

Part of #691 (the epic making every surface produce the same answer). The refusal it concerns
was introduced by #141 (enrolled PDFs yield no anchors, so the layout is declined rather than
answered wrongly).

Refs #691, #141

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions