Skip to content

Fix orphaned test false positives on behavioral/scenario test names#1121

Merged
chubes4 merged 2 commits intomainfrom
fix/orphaned-test-false-positives
Apr 11, 2026
Merged

Fix orphaned test false positives on behavioral/scenario test names#1121
chubes4 merged 2 commits intomainfrom
fix/orphaned-test-false-positives

Conversation

@chubes4
Copy link
Copy Markdown
Member

@chubes4 chubes4 commented Apr 11, 2026

Summary

Fixes #1120 — the orphaned test detector was deleting valid tests via autofix.

Root cause: The behavioral heuristic in find_orphaned_test_methods() used first-word matching that was too loose. Common verbs like apply, resolve, build matched source methods with those prefixes, causing scenario test names to be incorrectly flagged as orphaned.

Example from PR #1119: Test apply_replace_text (a scenario test for apply_edit_ops_to_content()) was flagged because:

  1. Strip test_apply_replace_text
  2. No exact source method match
  3. Behavioral filter: first word apply matches source method apply_edit_ops → not skipped
  4. Flagged as orphaned → autofix deleted it

Fix: Replace first-segment matching with direct-prefix matching. 3+ segment test names are now treated as behavioral (skipped) UNLESS the full expected source name is a direct prefix of a source method name. This is more conservative — some real orphans with 3+ segments may not be caught, but we stop deleting valid tests.

Changes

  • test_coverage.rs: Updated behavioral heuristic in find_orphaned_test_methods()
  • Updated existing test expectations (inline orphan test now expects 1 finding instead of 2)
  • Added regression test scenario_test_names_not_flagged_as_orphaned covering the exact PR feat: Add apply logic for EditOp (Phase 1 of #1041) #1119 scenario

Test results

22/22 test_coverage tests pass. Full suite: 1087 tests, 0 failures.

The behavioral heuristic in find_orphaned_test_methods() used first-word
matching that was too loose — common verbs like 'apply', 'resolve',
'build' matched source methods with those prefixes, causing scenario
tests to be flagged as orphaned and auto-deleted.

Replace first-segment matching with direct-prefix matching: 3+ segment
test names are only flagged if the full expected source name is itself a
prefix of a source method. This prevents false positives like
'apply_replace_text' being flagged because 'apply' matched 'apply_edit_ops'.

Fixes #1120
@homeboy-ci
Copy link
Copy Markdown
Contributor

homeboy-ci Bot commented Apr 11, 2026

Homeboy Results — homeboy

Lint

Failure Digest

Lint Failure Digest

Autofixability classification

  • Overall: auto_fixable
  • Autofix enabled: yes
  • Autofix attempted this run: no
  • Auto-fixable failed commands:
    • lint
  • Failed commands with available automated fixes:
    • lint

Machine-readable artifacts

  • {command}.json — structured output per command (from homeboy --output)

⚡ Scope: changed files only

lint (changed files only)

Audit

Failure Digest

Audit Failure Digest

  • Alignment score: 0.974
  • Severity counts: unknown: 1, warning: 7
  • Outliers in current run: 1
  • Parsed outlier entries: 1
  • Drift increased: yes
  • New findings since baseline: 4
    1. structural — File has 1103 lines (threshold: 1000) (structural::src/core/engine/edit_op_apply.rs::GodFile)
    2. test_coverage — Method 'apply_edit_ops' has no corresponding test (expected 'test_apply_edit_ops') (test_coverage::src/core/engine/edit_op_apply.rs::MissingTestMethod)
    3. test_coverage — Method 'apply_edit_ops_to_content' has no corresponding test (expected 'test_apply_edit_ops_to_content') (test_coverage::src/core/engine/edit_op_apply.rs::MissingTestMethod)
    4. test_coverage — Method 'resolve_anchor' has no corresponding test (expected 'test_resolve_anchor') (test_coverage::src/core/engine/edit_op_apply.rs::MissingTestMethod)
  • Top actionable findings:
    1. src/core/code_audit/test_coverage.rs — god_file — File has 1264 lines (threshold: 1000)
    2. src/core/engine/edit_op_apply.rs — god_file — File has 1103 lines (threshold: 1000)
    3. src/core/code_audit/test_coverage.rs — intra_method_duplicate — Duplicated block in analyze_test_coverage — 5 identical lines at line 89 and line 200
    4. src/core/code_audit/test_coverage.rs — missing_test_method — Method 'analyze_test_coverage' has no corresponding test (expected 'test_analyze_test_coverage')
    5. src/core/engine/edit_op_apply.rs — missing_test_method — Method 'apply_edit_ops' has no corresponding test (expected 'test_apply_edit_ops')
All parsed audit findings (8)
1. **src/core/code_audit/test_coverage.rs** — god_file — File has 1264 lines (threshold: 1000)
2. **src/core/engine/edit_op_apply.rs** — god_file — File has 1103 lines (threshold: 1000)
3. **src/core/code_audit/test_coverage.rs** — intra_method_duplicate — Duplicated block in `analyze_test_coverage` — 5 identical lines at line 89 and line 200
4. **src/core/code_audit/test_coverage.rs** — missing_test_method — Method 'analyze_test_coverage' has no corresponding test (expected 'test_analyze_test_coverage')
5. **src/core/engine/edit_op_apply.rs** — missing_test_method — Method 'apply_edit_ops' has no corresponding test (expected 'test_apply_edit_ops')
6. **src/core/engine/edit_op_apply.rs** — missing_test_method — Method 'apply_edit_ops_to_content' has no corresponding test (expected 'test_apply_edit_ops_to_content')
7. **src/core/engine/edit_op_apply.rs** — missing_test_method — Method 'resolve_anchor' has no corresponding test (expected 'test_resolve_anchor')
8. **src/commands/docs.rs** — outlier — (outlier)
- Full audit log: https://github.com/Extra-Chill/homeboy/actions/runs/24284394741

Autofixability classification

  • Overall: auto_fixable
  • Autofix enabled: yes
  • Autofix attempted this run: no
  • Auto-fixable failed commands:
    • audit
  • Failed commands with available automated fixes:
    • audit

Machine-readable artifacts

  • {command}.json — structured output per command (from homeboy --output)

⚡ Scope: changed files only

audit (changed files only)

Auto-refactor

ℹ️ Autofix enabled, but no fixable file changes were produced

Failure Digest

Autofixability classification

  • Overall: human_needed
  • Autofix enabled: yes
  • Autofix attempted this run: no
  • Human-needed failed commands:
    • refactor --from all

Machine-readable artifacts

  • {command}.json — structured output per command (from homeboy --output)

⚡ Scope: changed files only

refactor --from all

  • Total fixes proposed: 0
  • Files modified: 1
  • Stages:
    • audit: 0 fix(es), 1 file(s), 7 findings detected
    • lint: 0 fix(es), 0 file(s), 0 findings detected
    • test: 0 fix(es), 0 file(s)
Changed files (1)
1. `src/core/code_audit/test_coverage.rs`
Warnings (1)
Inter-stage formatter (sh '/home/runner/.config/homeboy/extensions/rust/scripts/format.sh') exited non-zero (continuing)
Tooling versions
  • Homeboy CLI: homeboy 0.88.7+cf82d0d5
  • Extension: rust from https://github.com/Extra-Chill/homeboy-extensions
  • Extension revision: unknown
  • Action: Extra-Chill/homeboy-action@v2

Homeboy Action v1

@chubes4 chubes4 merged commit e38ddc6 into main Apr 11, 2026
3 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OrphanedTest false positives: behavioral test names misidentified as method references

1 participant