You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Part of #1239. The contributor documentation describes how to organize tests, not which tests to write. Add a section that states which tests the suite needs and which it does not, and make the review skills check added and removed tests against it, so that the fixes in #1240–#1246 and later changes follow one written rule.
Facts
docs/testing.md "Organize tests" and the "Test Conventions" in AGENTS.md cover classes versus functions, parametrization, expected values, and IDs. Neither says what a test must protect.
docs/testing.md:233 says to retain "equivalent synchronous and asynchronous scenarios where applicable". The audit in Audit the test suite for unnecessary and flaky tests #1239 found that async and aio copies of sync tests that run through the same code are duplicates.
.agents/skills/self-review/SKILL.md asks whether a regression test "asserts observable behavior". No skill asks whether a test duplicates another or whether a removed test leaves its contract unprotected.
Proposed change
Add a "Choose what to test" section to docs/testing.md:
a test protects a caller-observable contract that no other test protects;
check behavior against real Athena and S3 where they can observe it, with only the queries and requests the contract needs;
mocked offline tests are suitable for complex logic tried in several patterns, such as concurrency, failure and abort combinations, and interrupts;
Activity