Validate endpoint test responses against the OpenAPI schema #131073

Endpoint tests should check each response against the published OpenAPI schema, both its status and its body. A test would then fail when an endpoint returns a status its @extend_schema doesn't list, or a body that no longer matches the documented shape. Requested by @BYK.

Why

In #131067, the Seer Autofix start endpoint (startOrganizationIssueAutofix) returned 402s and 409s with machine-readable code values, but its responses= listed neither status. @sentry/api is generated from this schema, so it typed those errors as unknown. The Sentry CLI and MCP in getsentry/toolkit map those codes to user-facing messages, so they had to hardcode them (getsentry/toolkit#2207). Every test for those paths passed the whole time, because nothing compares what an endpoint actually returns with what its schema says.

What exists today

• fixtures/apidocs_test_case.py: APIDocsTestCase.validate_schema() already checks a response against tests/apidocs/openapi-derefed.json with openapi_core. Tests have to opt in, it covers about 64 hand-written tests under tests/apidocs/endpoints/, and it asserts 200 <= status < 300, so error responses are never checked.
• tests/apidocs/test_response_coverage.py checks that 2xx fields have descriptions and types. It never looks at runtime responses.
• Static checks: the S034 flake8 rule (feat(apidocs): Add S034 lint for undocumented inline 4xx responses #131072) and the mypy plugin tightening (in progress separately).

Sketch

1. Hook the responses that API test cases already produce: APITestCaseMixin.get_response / get_success_response / get_error_response, or a pytest fixture that wraps the test client.
2. Resolve each response to its operation through the URL resolver (route name → path template → method + path in the derefed schema). Only check operations in the schema whose endpoint has publish_status[METHOD] == PUBLIC.
3. Check that:
   • the status is a key in that operation's responses. An undocumented 4xx is exactly the docs(seer): Document Autofix 402/409 error responses #131067 case.
   • the body validates against that status's schema, using the openapi_core validator APIDocsTestCase already uses, with the Spec loaded once per process.
4. Roll out as a ratchet, the way response-coverage-baseline.txt works. Record current mismatches in a committed baseline keyed by operation and status, fail only on new ones, and prune entries as they're fixed.
5. Ship report-only first (warn and collect), then make it fail once the baseline is stable.

Known limits

• It only sees what tests exercise. A status that no test triggers stays unchecked, so this complements the static checks rather than replacing them.
• It needs a fresh openapi-derefed.json (make build-api-docs). CI would have to build it before the backend tests or ship it as an artifact, and a local run with a stale file gives stale results.
• Generic error constants such as RESPONSE_BAD_REQUEST have no body schema, so for those statuses only the status itself is checked. Typed error bodies, like the ones docs(seer): Document Autofix 402/409 error responses #131067 added, get full checks.
• Performance: the validator is not free. Load the schema once, validate only public operations, and maybe sample 2xx validation if the suite slows down noticeably.
• Some endpoints return non-JSON responses (streams, redirects, files). Skip a response when the schema doesn't document a JSON body for it.

How it fits with the other checks

• S034 (flake8, feat(apidocs): Add S034 lint for undocumented inline 4xx responses #131072): fails at lint time when a method returns a 4xx Response(..., status=...) inline that isn't a key in responses=. It's cheap and catches the common case while the code is being written, but it can't follow helpers, variables or raised exceptions.
• mypy plugin tightening: checks literal error bodies against their typed response arms at type-check time, for bodies built inline.
• This check: catches whatever gets past both at runtime: statuses from helpers and exceptions, and bodies assembled dynamically. That's the only one of the three that sees what the endpoint actually returns.

via @BYK.

--

View Session [Sentry]