[sentry-junior]

Proof of concept for #131073: find out what it costs to check endpoint test responses against the published OpenAPI schema. It's opt-in and changes nothing unless you set the env var.

make build-api-docs
SENTRY_OPENAPI_RESPONSE_CHECK=report pytest tests/sentry/seer/endpoints/test_group_ai_autofix.py

What it does

A pytest plugin (sentry.testutils.pytest.openapi_responses) wraps django.test.Client.request, so it sees every request a test makes: get_success_response, get_error_response, and plain self.client.get(...) too. For each response it:

1. Matches method + path against the paths in tests/apidocs/openapi-derefed.json. Literal segments win over {params}. A path or method that isn't in the schema isn't public and is skipped (internal RPC, private endpoints).
2. Reports undocumented-status when the status isn't a key in that operation's responses.
3. Reports body-mismatch when the status has a JSON schema and the rendered body doesn't validate against it (openapi_schema_validator.OAS30ReadValidator, cached per operation and status).

Each xdist worker writes its findings to .artifacts/openapi-response-check/<worker>.json, and the controller prints one deduplicated summary. report only collects findings. strict also raises in the request that produced the mismatch.

This PR has no baseline or ratchet and no CI wiring yet. Those depend on the cost numbers below.

Cost

Measured on an 8 vCPU sandbox:

make build-api-docs (deprecated + spectacular --validate + deref)	~27s (spectacular without --validate: 16s, deref: 6s)
Loading the 16 MB derefed spec and indexing 187 paths / 289 ops, per worker	0.2–0.3s
tests/sentry/core/endpoints + tests/sentry/issues/endpoints, -n 6, ~1,600 tests	4.05s total check time across 6 workers for 20,549 requests (1,562 matched to public ops). That's under 0.5% of test CPU.
test_group_ai_autofix.py, 889 requests	0.10s

The wall-clock difference between runs with and without the check was within noise. Snuba-backed tests in this sandbox flake differently from run to run under xdist. Run serially, the same files pass identically with the check on and off.

The check itself is close to free. The real cost is getting a fresh openapi-derefed.json to wherever the tests run.

Findings from the sample run

These come from two test directories plus the autofix tests, and include the endpoint that #131067 fixed:

• POST …/issues/{issue_id}/autofix/ 202: the coding_agent_handoff step returns AutofixHandoffResponse (successes/failures), but the schema documents only AutofixPostResponse ('feature_id' is a required property). This is the same kind of drift as docs(seer): Document Autofix 402/409 error responses #131067.
• GET …/autofix/ 304: the conditional GET isn't documented.
• Body drift: listOrganizations/getProject allowMemberInvite is missing, listOrganizationIssues inbox and updateOrganizationIssues assignedTo return null but aren't nullable, getOrganizationMember user.avatarUrl is missing, updateProjectOwnership owners[].name is missing, listOrganizationIssueEvents event.type is missing, listOrganizationIssueTagValues key is missing, listTeamMembers user is null.
• Undocumented statuses: deleteProject 401 (sudo), updateOrganizationMember 404, listOrganizationTeams 400, updateProjectOwnership 403, updateOrganizationIssue 204 (discard), getProject 302 (renamed project).
• Noise that needs a policy decision: 429 from rate-limit tests and 5xx from mocked timeouts. These probably need one global exemption, or a single shared documented response.

That's 24 distinct mismatches from about 1,600 tests.

Next steps (not in this PR)

• Baseline/ratchet keyed by METHOD template status kind, like response-coverage-baseline.txt.
• CI: the api-docs job already builds the spec on every backend PR (its filter includes backend_all), so the cheapest wiring is to upload openapi-derefed.json as an artifact there and set SENTRY_OPENAPI_RESPONSE_CHECK=report in the backend-test shards that download it. Alternatively, generate the spec inside the pytest controller (~16s per shard, always fresh).

--

View Session [Sentry]