Skip to content

pr_runtime: group Go subtest failures under parents - #127

Open
punithk-verse wants to merge 2 commits into
huggingface:mainfrom
punithk-verse:fix/go-subtest-untracked-failures
Open

punithk-verse wants to merge 2 commits into
huggingface:mainfrom
punithk-verse:fix/go-subtest-untracked-failures

Conversation

@punithk-verse

Copy link
Copy Markdown

summary

  • Group failed Go subtests under their represented parent when calculating untracked_failed.
  • Prevent Go subtests from being counted as separate untracked failures.
  • Keep non-Go runner behavior unchanged.

Test plan

  • uv run pytest -q tests/test_pr_runtime_verifier.py — 22 passed
  • uv run ruff check src/repo2rlenv/pipelines/_pr_runtime_verifier.py tests/test_pr_runtime_verifier.py
  • git diff --check

Out of scope

  • changes to the canonical Go log parser
  • changes to non-Go grading behavior
  • changes to runtime test execution

@KNambiarDJsc KNambiarDJsc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed and tested 0b02021:

  • Unit tests: tests/test_pr_runtime_verifier.py: 22 passed. ruff check / ruff format --check: clean.
  • runner is additive and keyword-defaulted (runner: str | None = None), so every existing non-Go call site keeps exactly the old untracked_failed computation — verified by reading through, not just trusting the default.
  • Ran my own adversarial probes against grade() directly, beyond what's in the PR:
    • A same-named-but-unrelated top-level test (Foo: PASSED) doesn't falsely swallow an unrelated Bar/Foo: FAILED — the ancestor walk is exact-segment, not substring, so no accidental collapsing.
    • The realistic production shape — a tracked parent (TestFeature in fail_to_pass) with untracked failing subtests underneath — correctly suppresses the subtests from untracked_failed while still scoring the parent normally through f2p_passed. This is the actual motivating case (subtest names aren't knowable ahead of time to put in F2P/P2P) and it holds up.
    • Deep nesting where only a grandparent (not the immediate parent) appears in status_map still collapses correctly to the single topmost tracked ancestor.
    • Non-Go runner path with slash-containing test names is untouched, confirming the runner gate isn't accidentally broad.

LGTM.

@adithya-s-k

Copy link
Copy Markdown
Collaborator

Thanks, tested this and found one case to fix before merging: a failed child is dropped whenever its parent exists, even if that parent is PASSED. This can happen when two Go packages use the same test name. Please only group under an ancestor that actually records the failure, and add a regression test. The graded reward is unchanged, but we shouldn't lose the failure from the diagnostics.

@punithk-verse

Copy link
Copy Markdown
Author

Implemented in commit 501902e.

  • Updated _has_go_parent() so a Go subtest is grouped under an ancestor only when that ancestor is recorded as FAILED, rather than merely being present in status_map.
  • Added a regression test covering the case where the parent test is PASSED but its subtests FAILED; the failed subtests are now retained in untracked_failed instead of being incorrectly swallowed.
  • Updated the main() integration test to verify the same behavior end-to-end.
  • Verified with 22 passed and Ruff clean.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants