Skip to content

Fix Execution API JWT reissue TOCTOU race - #70227

Open
anmolxlight wants to merge 1 commit into
apache:mainfrom
anmolxlight:fix-jwt-reissue-mid-request
Open

Fix Execution API JWT reissue TOCTOU race#70227
anmolxlight wants to merge 1 commit into
apache:mainfrom
anmolxlight:fix-jwt-reissue-mid-request

Conversation

@anmolxlight

Copy link
Copy Markdown
Contributor

closes: #70222

Problem

Intermittent ExpiredSignatureError / Failed to validate JWT on /execution/... for tasks that are not necessarily older than [execution_api] jwt_expiration_time.

Root cause is a TOCTOU race in JWTReissueMiddleware:

  1. JWTBearer validates the bearer token (still valid) and caches claims on the ASGI request scope.
  2. The request handler runs (e.g. heartbeat).
  3. Middleware re-calls avalidated_claims on the same bearer string.
  4. If the token crossed exp during the request, that second validation raises ExpiredSignatureError.
  5. The exception is swallowed, so no Refreshed-API-Token header is returned.
  6. The client's next call arrives with a fully expired token → 403, and after enough failed heartbeats the task is killed.

This matches the intermittent multi-task failures reported in #70222 and the long-running LocalExecutor case in #67939.

Fix

Reuse the claims already validated and cached by JWTBearer (request.scope["ti_token"]) instead of re-parsing/re-validating the Authorization header. If remaining lifetime is below the existing refresh threshold, issue a fresh token from those claims.

No expiry leeway is added. Workload tokens remain non-refreshable.

Tests

  • Updated threshold reissue tests to populate the ASGI scope the way production does.
  • Added a mid-request expiry regression test (custom route advances the clock past exp during the handler) asserting Refreshed-API-Token is still issued and avalidated_claims is not called by middleware.

Local verification

uv run --project airflow-core pytest airflow-core/tests/unit/api_fastapi/execution_api/versions/head/test_router.py -xvs
ruff check airflow-core/src/airflow/api_fastapi/execution_api/app.py airflow-core/tests/unit/api_fastapi/execution_api/versions/head/test_router.py
ruff format --check airflow-core/src/airflow/api_fastapi/execution_api/app.py airflow-core/tests/unit/api_fastapi/execution_api/versions/head/test_router.py

All 5 tests passed.

Notes

Related open PR #68499 targets the same race for #67939. This PR is based on current main, includes ashb's requested mid-request time-travel regression shape, and closes #70222.

JWTReissueMiddleware re-validated the bearer token after the handler ran.
If the token expired mid-request, ExpiredSignatureError was swallowed and
no Refreshed-API-Token was returned, so the next heartbeat failed with 403.

Reuse claims already cached on the ASGI scope by JWTBearer instead.

Closes: apache#70222
Related: apache#67939
Signed-off-by: anmolxlight <anmolx.work@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:API Airflow's REST/HTTP API area:task-sdk ready for maintainer review Set after triaging when all criteria pass.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Execution API JWT expires unexpectedly in Airflow 3.2.2 (ExpiredSignatureError on /execution/...)

2 participants