-
Notifications
You must be signed in to change notification settings - Fork 0
Description
Summary
Based on production usage in MCP Moira project, several improvements would make testfold more agent-friendly:
1. Clear Agent Instructions on Failure
When tests fail, output explicit instructions for AI agents:
┌─────────────────────────────────────────────────────────────┐
│ AGENT INSTRUCTIONS: │
│ │
│ 1. Read .md files above - they contain full error info │
│ 2. DO NOT re-run tests - artifacts will be overwritten │
│ 3. DO NOT read .log files - use .md reports instead │
│ 4. Fix issues based on failure reports, then re-run tests │
└─────────────────────────────────────────────────────────────┘
Why: Agents often waste tokens re-running tests or reading full logs when failure reports already exist.
2. List Exact Failure Report Paths
Instead of just showing the failures directory, list each .md file:
Unit (2 failed):
📄 test-results/failures/unit/01-auth-login-should-validate-token.md
📄 test-results/failures/unit/02-user-service-should-create-user.md
Why: Agent can immediately read specific files without listing directory.
3. Test File Filter with Auto-Skip
When running with a test file argument, suites that don't match should show "Skipped" not "Failed":
testfold -- auth.test.tsOutput:
⊘ Integration Tests Skipped (no matching files)
⊘ E2E Tests Skipped (no matching files)
✓ Unit Tests: 15 passed (2.1s)
Current behavior: "No tests found" is treated as failure.
4. Glob-Based Test File Resolution
Allow passing just filename and resolve it recursively:
testfold -- user-service.test.ts
# Resolves to: tests/unit/services/user-service.test.tsShow resolution in output:
🔍 Resolved test file to: tests/unit/services/user-service.test.ts
Why: Agents don't always know full path structure.
5. Failure Marker in Artifacts Table
Mark failed suites clearly in artifacts summary:
📂 All artifacts: test-results/
Unit unit.json | unit.log | failures/unit/ ◀ READ THIS
E2E e2e.json | e2e.log | failures/e2e/
Implementation Notes
These changes are already implemented and tested in MCP Moira's test runner (tests/run-all-tests.js). Can provide reference implementation.