Skip to content

Document which tests the suite needs #1247

Closed

Description

@laughingman7743

Summary

Part of #1239. The contributor documentation describes how to organize tests, not which tests to write. Add a section that states which tests the suite needs and which it does not, and make the review skills check added and removed tests against it, so that the fixes in #1240–#1246 and later changes follow one written rule.

Facts

  • docs/testing.md "Organize tests" and the "Test Conventions" in AGENTS.md cover classes versus functions, parametrization, expected values, and IDs. Neither says what a test must protect.
  • docs/testing.md:233 says to retain "equivalent synchronous and asynchronous scenarios where applicable". The audit in Audit the test suite for unnecessary and flaky tests #1239 found that async and aio copies of sync tests that run through the same code are duplicates.
  • .agents/skills/self-review/SKILL.md asks whether a regression test "asserts observable behavior". No skill asks whether a test duplicates another or whether a removed test leaves its contract unprotected.

Proposed change

  1. Add a "Choose what to test" section to docs/testing.md:
    • a test protects a caller-observable contract that no other test protects;
    • check behavior against real Athena and S3 where they can observe it, with only the queries and requests the contract needs;
    • mocked offline tests are suitable for complex logic tried in several patterns, such as concurrency, failure and abort combinations, and interrupts;
    • what to avoid: criteria 1–8 of Audit the test suite for unnecessary and flaky tests #1239;
    • test asynchronous and aio variants where they have their own code;
    • a test must fail when the behavior it names is removed, and every wait is bounded.
  2. Replace the sentence at docs/testing.md:233 to match.
  3. Add a pointer to the section from the "Test Conventions" in AGENTS.md.
  4. Add checks for added and removed tests to the self-review, self-review-round-two, and independent-review skills.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions