Skip to content

fix: reject JUnit testcases without a name instead of merging them into one check - #97

Merged
noteflowai merged 1 commit into
mainfrom
automation/feature-2026-11-05-evalarc-134123710674
Nov 7, 2026
Merged

noteflowai merged 1 commit into
mainfrom
automation/feature-2026-11-05-evalarc-134123710674

Conversation

@noteflowai

Copy link
Copy Markdown
Owner

EA-02 requires that malformed records and duplicate IDs cannot look like an improvement or an unchanged result. The JUnit loader in src/evalarc/results_diff.py builds a case ID from the testcase name attribute without checking that the attribute exists. A with no name becomes 'classname::None', or 'None' when no owner is known. One with an empty name becomes 'classname::'. Every such testcase in a suite is merged into one check and counted as a repeated attempt. Traced through _tally and _classify, this hides a swapped failure. A baseline of [fail, pass] and a current run of [pass, fail] both tally 1/2, so the check is classified as unchanged and diff exits 0. Inspect and promptfoo inputs with missing or duplicate identity already exit 2: an Inspect sample with no integer or string id, a repeated promptfoo result row. JUnit has no equivalent check, and neither the EA-02 corpus nor tests/test_results_diff.py contains an unnamed JUnit testcase.

Scope: only the JUnit branch of load_results, reached through _junit(raw) in src/evalarc/results_diff.py, with evalarc diff as the entry point under test.

Input: a JUnit XML document already accepted by the existing DTD, parse and root-tag checks. For each processed by the existing iteration (the root itself, or every testsuite found under ), and for each direct child in document order, the loader reads the name attribute before building the case ID.

New validation: if name is missing, or is empty or whitespace-only after stripping, loading raises ValueError with this message:
JUnit testcase at position P in suite S has no name attribute; each testcase needs a name to be compared. Re-export the results with test names or fix the XML.
P is the 0-based index of that testcase among the direct testcase children of its suite. S is the suite name rendered with repr(), for example 'corpus'. When the suite has no name, S is the text (unnamed suite). The first offending testcase in document order is reported and loading stops. No partial run is returned.

evalarc diff handles this ValueError through its existing unusable-input path, the same one test_unusable_inputs_exit_2 exercises for the DTD error. The command exits 2 and prints the message on stderr. The check applies to whichever file, baseline or current, contains the unnamed testcase.

Unchanged behavior:

  • Named testcases keep their exact case IDs (classname::name, suite::name, or the bare name when there is no owner). Their verdicts, evidence text, usage records, identity suites and headline do not change.
  • Two testcases with the same non-empty classname and name still load as repeated attempts of one check, as docs/ci-gate.md documents for repeated JUnit cases.
  • A name that is non-empty after stripping is used verbatim, without stripping, so existing IDs do not change.
  • DTD/entity rejection, parse errors, root-tag errors, the no-scored-cases error, Inspect and promptfoo loading, diff classification, exit codes 0 and 1, and all report output are unchanged.

Documentation: in docs/ci-gate.md, add 'a JUnit testcase without a name' to the exit-2 list. Add one short paragraph that explains the rejection (such a testcase cannot be paired across runs and would merge with other unnamed testcases into a single check) and the recovery step. Make the matching change in docs/ci-gate.zh-CN.md.

Acceptance test file: tests/test_feature_5e40b6f9743a.py. It is labeled SYNTHETIC, writes hand-written JUnit XML in tmp_path, and calls load_results, diff and evalarc.cli main the same way the existing EA-02 corpus does.

Acceptance:

  • Baseline suite 'corpus' has two classname='corpus' testcases without a name attribute (the first with , the second passing). The current file has the outcomes swapped. evalarc diff exits 2 (previously 0), and stderr contains "JUnit testcase at position 0 in suite 'corpus' has no name attribute".
  • A testcase with name="" or name=" " at position 1 of suite 'corpus' makes load_results raise ValueError containing "JUnit testcase at position 1 in suite 'corpus' has no name attribute". evalarc diff exits 2 whether that file is the baseline or the current input.
  • An unnamed testcase inside a without a name, under a root, raises a ValueError containing 'in suite (unnamed suite) has no name attribute'.
  • Control and compatibility: the same two-testcase files with names test_a and test_b load as 'corpus::test_a' and 'corpus::test_b', and the swapped outcomes give one regressed and one improved change with exit 1. Two testcases with identical classname='corpus' name='test_a' still load as one check with two attempts. tests/test_results_diff.py and tests/test_feature_cb4fe5a9ab91.py pass unchanged.
  • docs/ci-gate.md and docs/ci-gate.zh-CN.md list a JUnit testcase without a name among the exit-2 causes and give the recovery step.

One specialist feature. Behavioral acceptance fails on the base and passes on the implementation; project checks and independent model review passed. GPU results are included only when actually executed. Automatically developed and reviewed; no human review is claimed.

@noteflowai
noteflowai merged commit c5d3e9d into main Nov 7, 2026
15 checks passed
@noteflowai
noteflowai deleted the automation/feature-2026-11-05-evalarc-134123710674 branch November 7, 2026 18:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant