Skip to content

Epic: the same two files should produce the same answer on every surface #691

Description

@willhea

Problem

DeltaTrack can be asked to compare two bill versions in three ways: the diff_bill.py and
diff_pdf.py commands, the POST /api/compare web endpoint behind the upload page, and
scripts/render_examples.py, which produces the reports published as the project's public
examples. For the same two files, those surfaces do not all give the same answer.

The good news first, because it bounds this epic. The diff engine itself is genuinely shared,
and the sharing holds. Measured on the committed fixture pair tests/corpus/118-hr-8752
(1_reported-in-house to 2_engrossed-in-house), a report rendered by the command line and a
report rendered by the web endpoint differ by two lines out of roughly nine hundred. Every
change card, every word-level diff and every line of the full-bill view is byte-identical. The
consolidation work behind #42 (one module assembles a bill-XML report), #367 (the engine does
not depend on a delivery channel), ADR 0006 (the canonical diff contract) and ADR 0007 (a single
HTML renderer) did the hard part.

What still diverges sits in a thin layer above the engine, and it is small enough to enumerate
exactly:

  1. Version identity. Turning a filename into a version label and an ordinal is
    re-implemented at six places under two different algorithms, so one PDF pair renders three
    different report headings depending on who asked.
  2. The JSON contract. --format json on the command line returns the engine's internal
    dictionary while ?output=json on the endpoint returns the published canonical document.
    They share two top-level keys.
  3. Error presentation. A document the engine deliberately declines reaches the web user as a
    written explanation and the command-line user as a traceback.

None of it is protected by a test, which is why all three accumulated quietly.

Why it matters

The three surfaces are read as one product. A reader who opens a published example and then runs
the same comparison locally is entitled to the same document; today the heading differs. Someone
building against the output has to know which surface produced it before they can parse it. And
the divergence is invisible to CI, so it grows rather than shrinks.

None of this is breaking anything today, which is the argument for doing it now rather than under
pressure. Each item is a small, self-contained change; the reason to group them is that they
share one root cause, and fixing them individually without the gate would let them drift back.

Decomposition

Each sub-issue is closeable by its own verifiable change.

Sub-issue What it settles State
#692, three different version headings for one bill pair One resolver from a source document to its label and ordinal, called by every surface Ready
#693, the command line returns internal JSON, not the contract Deletes the vestigial output so both surfaces return the published contract Ready, step 1
#698, the XML pipeline routes through an intermediate dictionary Removes the redundant representation, so both pipelines take the route the PDF one already does Step 2, after #653
#695, a declined document reaches the command line as a traceback The engine's user-facing refusal is presented as a message on every surface Ready
#696, nothing checks that the two surfaces agree The gate that holds all of the above Blocked on the above

The JSON work is deliberately two steps. Step 1 (#693), now: delete the command line's
internal JSON output, so --format json returns the canonical document and both surfaces answer
the same question the same way. Step 2 (#698), later: remove the intermediate dictionary the
XML pipeline builds on the way to that document. Step 2 follows #653 (make the view a consumer
of the diff, not a second producer of it), which is reshaping the same boundary; doing it first
means doing it twice.

Two questions are settled and no longer tracked separately.

  • --financial filters on one output format and both filters and enriches on the other #694 (--financial filters on one output format and both filters and enriches on the other) is
    closed as superseded. The split lives entirely inside the branch step 1 deletes.
  • Making the canonical document able to carry unchanged nodes was considered and deferred. No
    current consumer needs to ask which parts of a bill did not change. --include-unchanged is
    removed in step 1 along with the output it was the only meaningful consumer of, and the
    capability returns as an additive schema change if a need appears. The cost of the two possible
    shapes was measured first, in docs/research/cli-ui-parity/probes/probe_unchanged.py: carrying
    unchanged entries in the changes array more than doubles a near-identical comparison, while
    giving each structure-tree node an id costs a fixed few kilobytes.

The parity gate (#696) lands last, once the surfaces it pins have stopped moving.

One observation that is not filed as a sub-issue

No capability is available on all three surfaces. diff_bill.py compare has --filter and
--financial; diff_pdf.py has neither, nor a JSON output, but is the only surface with
explicit --v1-label / --v2-label flags; the endpoint has none of them. Some of those gaps are
defensible (filters may genuinely not belong on a public endpoint) and some are accretion.
Recorded here as context for whoever picks up the sub-issues rather than filed, because "make the
matrix square" is not a verifiable change and the individual answers belong with the surfaces
they affect.

How it surfaced

An audit of the two diff paths on develop at commit 8fb0fa16, driving the real entry points
rather than reading the source: argparse main() for the commands, and the FastAPI route
through Starlette's TestClient for the endpoint. The probes are on branch
worktree-audit-cli-ui-diff-parity under docs/research/cli-ui-parity/, and each sub-issue
names the one that produced its evidence.

Checked for overlap

These findings were checked against the work already in flight and are not covered by it. #676
(the epic making DeltaTrack's own identity independent of BillTrax) covers the design tokens, the
decision records and three developer scripts; #653 (make the view a consumer of the diff) covers
the renderer re-deriving what the diff already settled; #689 (the report names the same concept
two ways) covers CSS class vocabulary against the canonical contract; #688 (decide DeltaTrack's
palette in one module) covers styling. None of them touches how a filename becomes a version
identity, or what the two JSON outputs contain. All twelve open pull requests were checked as
well.

The project has no external consumers it is currently supporting, which is why step 1 deletes
rather than deprecates.

How this sits against the other epics

Checked against all fourteen open epics. Four relationships are load-bearing enough to record here
rather than leaving to discovery.

Epic Relationship
#228, support XML-less bill versions via the HTML rendition Order matters. It adds a third input pipeline, which becomes a seventh version-identity derivation site (#692) and a third route to the canonical document whose shape #698 is settling. That epic states it is not urgent; this one is small and near-term, so the cheap ordering is to consolidate first. Noted on #228.
#653, make the view a consumer of the diff Blocks step 2. #698 rebuilds the XML adapter; #653 is reshaping what that adapter produces. The audit also answered one of #653's stated unknowns: the XML path has no second document, but it does discard element_id after using it, which is the same rule violated a different way. Noted on #653.
#552, container levels and node addresses derived from display labels Owns a deferred question. Item 3 of that epic, giving a node a structural address independent of its printed label, is the prerequisite for the canonical document being able to say which nodes did not change. That capability was measured and deferred here; if a consumer needs it, it belongs to #552 rather than to a fresh schema issue.
#288, the test suite fails open #696 is an instance of its thesis. Four surface divergences accumulated on a checkout where 2028 tests pass. The parity gate is the specific missing check; #299 (no standing way to prove a test can fail) is what its verification depends on.

Two more do not change the work but change how it is read.

#360 (report generation costs more than required) constrains the deferred unchanged-nodes
option. Carrying unchanged entries in the changes array would have added up to 113% to a document
already embedded in every report, which is the opposite direction from that epic. It reinforces the
deferral and, if the capability returns, argues for the node-id shape over the entry-list one.

#555 (the develop to main promotion is unowned) amplifies the user-facing half of #692 without
changing its fix. The published site builds from main, which is hundreds of commits behind, and
#451 records that the published example reports are frozen there too. The three-way heading
disagreement in #692 was measured with all three surfaces at develop, so it is a real code defect
independent of deployment; but a visitor comparing the live site's example against their own run
today sees a larger gap than the one measured, and the fix reaches them only at the next promotion.

Checked and found not to overlap: #676 (BillTrax's privileged standing, near complete), #689 and
#688 (CSS vocabulary and palette, the same one-concept-many-spellings family but a different layer),
#551, #298, #289, #3, #147, #175, #370.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    epicContainer of sub-issues; tracked on the Roadmap, not the working board

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions