Skip to content

feat: Agent-friendly output improvements #27

@witqq

Description

@witqq

Summary

Based on production usage in MCP Moira project, several improvements would make testfold more agent-friendly:

1. Clear Agent Instructions on Failure

When tests fail, output explicit instructions for AI agents:

┌─────────────────────────────────────────────────────────────┐
│  AGENT INSTRUCTIONS:                                        │
│                                                             │
│  1. Read .md files above - they contain full error info    │
│  2. DO NOT re-run tests - artifacts will be overwritten    │
│  3. DO NOT read .log files - use .md reports instead       │
│  4. Fix issues based on failure reports, then re-run tests │
└─────────────────────────────────────────────────────────────┘

Why: Agents often waste tokens re-running tests or reading full logs when failure reports already exist.

2. List Exact Failure Report Paths

Instead of just showing the failures directory, list each .md file:

Unit (2 failed):
  📄 test-results/failures/unit/01-auth-login-should-validate-token.md
  📄 test-results/failures/unit/02-user-service-should-create-user.md

Why: Agent can immediately read specific files without listing directory.

3. Test File Filter with Auto-Skip

When running with a test file argument, suites that don't match should show "Skipped" not "Failed":

testfold -- auth.test.ts

Output:

⊘ Integration Tests Skipped (no matching files)
⊘ E2E Tests Skipped (no matching files)
✓ Unit Tests: 15 passed (2.1s)

Current behavior: "No tests found" is treated as failure.

4. Glob-Based Test File Resolution

Allow passing just filename and resolve it recursively:

testfold -- user-service.test.ts
# Resolves to: tests/unit/services/user-service.test.ts

Show resolution in output:

🔍 Resolved test file to: tests/unit/services/user-service.test.ts

Why: Agents don't always know full path structure.

5. Failure Marker in Artifacts Table

Mark failed suites clearly in artifacts summary:

📂 All artifacts: test-results/
Unit       unit.json | unit.log | failures/unit/ ◀ READ THIS
E2E        e2e.json | e2e.log | failures/e2e/

Implementation Notes

These changes are already implemented and tested in MCP Moira's test runner (tests/run-all-tests.js). Can provide reference implementation.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions