Repository navigation
fix: reject JUnit testcases without a name instead of merging them into one check - #97
Merged
noteflowai merged 1 commit intoNov 7, 2026
Conversation
noteflowai
deleted the
automation/feature-2026-11-05-evalarc-134123710674
branch
November 7, 2026 18:35
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
EA-02 requires that malformed records and duplicate IDs cannot look like an improvement or an unchanged result. The JUnit loader in src/evalarc/results_diff.py builds a case ID from the testcase name attribute without checking that the attribute exists. A with no name becomes 'classname::None', or 'None' when no owner is known. One with an empty name becomes 'classname::'. Every such testcase in a suite is merged into one check and counted as a repeated attempt. Traced through _tally and _classify, this hides a swapped failure. A baseline of [fail, pass] and a current run of [pass, fail] both tally 1/2, so the check is classified as unchanged and diff exits 0. Inspect and promptfoo inputs with missing or duplicate identity already exit 2: an Inspect sample with no integer or string id, a repeated promptfoo result row. JUnit has no equivalent check, and neither the EA-02 corpus nor tests/test_results_diff.py contains an unnamed JUnit testcase.
Scope: only the JUnit branch of load_results, reached through _junit(raw) in src/evalarc/results_diff.py, with evalarc diff as the entry point under test.
Input: a JUnit XML document already accepted by the existing DTD, parse and root-tag checks. For each processed by the existing iteration (the root itself, or every testsuite found under ), and for each direct child in document order, the loader reads the name attribute before building the case ID.
New validation: if name is missing, or is empty or whitespace-only after stripping, loading raises ValueError with this message:
JUnit testcase at position P in suite S has no name attribute; each testcase needs a name to be compared. Re-export the results with test names or fix the XML.
P is the 0-based index of that testcase among the direct testcase children of its suite. S is the suite name rendered with repr(), for example 'corpus'. When the suite has no name, S is the text (unnamed suite). The first offending testcase in document order is reported and loading stops. No partial run is returned.
evalarc diff handles this ValueError through its existing unusable-input path, the same one test_unusable_inputs_exit_2 exercises for the DTD error. The command exits 2 and prints the message on stderr. The check applies to whichever file, baseline or current, contains the unnamed testcase.
Unchanged behavior:
Documentation: in docs/ci-gate.md, add 'a JUnit testcase without a name' to the exit-2 list. Add one short paragraph that explains the rejection (such a testcase cannot be paired across runs and would merge with other unnamed testcases into a single check) and the recovery step. Make the matching change in docs/ci-gate.zh-CN.md.
Acceptance test file: tests/test_feature_5e40b6f9743a.py. It is labeled SYNTHETIC, writes hand-written JUnit XML in tmp_path, and calls load_results, diff and evalarc.cli main the same way the existing EA-02 corpus does.
Acceptance:
One specialist feature. Behavioral acceptance fails on the base and passes on the implementation; project checks and independent model review passed. GPU results are included only when actually executed. Automatically developed and reviewed; no human review is claimed.