Airflow 3.2 Worker Failures on TaskInstance Finish (HTTP 409 invalid_state)

[DominikBalga]

Under which category would you file this issue?

Task SDK

Apache Airflow version

3.2.0. & 3.2.1

What happened and how to reproduce it?

Issue Description

The observed behavior starts after upgrading to Airflow 3.2.0 (and persists in 3.2.1) and disappears when downgrading to 3.1.8.

The failure manifests on Celery workers during the post-execution "finish" step (Task SDK / execution API), where the worker attempts to PATCH the TaskInstance state and receives:

• HTTP 409
• reason=invalid_state
• message=TI was not in the running state so it cannot be updated
• previous_state=success

This results in a worker-side exception (ServerResponseError) and causes the Celery workload to be treated as failed even when the underlying task logic has completed.

• Airflow version(s) affected: 3.2.0, 3.2.1
• Airflow version known-good: 3.1.8
• Executor: CeleryExecutor
• Deployment: Kubernetes
• Python: 3.12
• Database: Postgres

Typical worker log sequence:

• Worker starts and loads secrets backend
• Worker begins Task SDK API calls (often with retries)
• API returns 409 invalid_state with previous_state=success
• Supervisor raises ServerResponseError and the Celery task fails

Example:

• Secrets backends loaded for worker ...
• Starting call to 'airflow.sdk.api.client.Client.request', this is the 1st time calling it.
• Starting call to ... this is the 2nd time calling it.
• API server error ... status_code=409 ... previous_state='success'
• airflow.sdk.api.client.ServerResponseError: Server returned error

Steps to reproduce

The issue appears intermittently; higher concurrency tends to increase the frequency.

Prerequisites

• Airflow 3.2.0+
• CeleryExecutor
• Execution API enabled (Task SDK in use)

1. Deploy Airflow 3.2.0 or 3.2.1.
2. Trigger the DAG multiple times (or run multiple manual runs concurrently).
3. Observe worker logs for ServerResponseError with HTTP 409 invalid_state and previous_state=success.

What you think should happen instead?

Airflow workers should be able to finalise a task idempotently; if the API server has already moved the TaskInstance to a terminal state (e.g. success), the worker’s PATCH /execution/task-instances//state should not return a fatal 409 or fail the task.

Operating System

No response

Deployment

Official Apache Airflow Helm Chart

Apache Airflow Provider(s)

No response

Versions of Apache Airflow Providers

No response

Official Helm Chart version

Not Applicable

Kubernetes Version

No response

Helm Chart configuration

No response

Docker Image customizations

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[serhii-karelov]

We noticed the problem where heartbeat DB queries queue up. This makes workers timeout on the heartbeat request and retry. I can imagine that the failed heartbeat can put task instance into a terminal state (but not sure though).

usename    state                xact_duration    query_duration    query
db_user  idle in transaction  00:00:22.17721   00:00:00.06658    SELECT task_instance.state AS task_instance_state
                                                                   FROM task_instance
                                                                   WHERE task_instance.id = '0<zzzyyy>54'::uuid::UUID
db_user  active               00:00:22.123495  00:00:22.120761   UPDATE task_instance SET last_heartbeat_at='2026-04-23T09:03:18.258390+00:00'::timestamptz, updated_at='2026-04-23T09:03:18.261693+00:00'::timestamptz WHERE task_instance.id
db_user  active               00:00:22.058237  00:00:22.054038   UPDATE task_instance SET last_heartbeat_at='2026-04-23T09:03:18.320462+00:00'::timestamptz, updated_at='2026-04-23T09:03:18.324080+00:00'::timestamptz WHERE task_instance.id
db_user  active               00:00:21.901896  00:00:21.89822    UPDATE task_instance SET last_heartbeat_at='2026-04-23T09:03:18.479643+00:00'::timestamptz, updated_at='2026-04-23T09:03:18.483318+00:00'::timestamptz WHERE task_instance.id
db_user  active               00:00:21.537818  00:00:21.534485   UPDATE task_instance SET last_heartbeat_at='2026-04-23T09:03:18.844164+00:00'::timestamptz, updated_at='2026-04-23T09:03:18.847447+00:00'::timestamptz WHERE task_instance.id
...

[DominikBalga]

We noticed the problem where heartbeat DB queries queue up. This makes workers timeout on the heartbeat request and retry. I can imagine that the failed heartbeat can put task instance into a terminal state (but not sure though).

usename    state                xact_duration    query_duration    query
db_user  idle in transaction  00:00:22.17721   00:00:00.06658    SELECT task_instance.state AS task_instance_state
                                                                   FROM task_instance
                                                                   WHERE task_instance.id = '0<zzzyyy>54'::uuid::UUID
db_user  active               00:00:22.123495  00:00:22.120761   UPDATE task_instance SET last_heartbeat_at='2026-04-23T09:03:18.258390+00:00'::timestamptz, updated_at='2026-04-23T09:03:18.261693+00:00'::timestamptz WHERE task_instance.id
db_user  active               00:00:22.058237  00:00:22.054038   UPDATE task_instance SET last_heartbeat_at='2026-04-23T09:03:18.320462+00:00'::timestamptz, updated_at='2026-04-23T09:03:18.324080+00:00'::timestamptz WHERE task_instance.id
db_user  active               00:00:21.901896  00:00:21.89822    UPDATE task_instance SET last_heartbeat_at='2026-04-23T09:03:18.479643+00:00'::timestamptz, updated_at='2026-04-23T09:03:18.483318+00:00'::timestamptz WHERE task_instance.id
db_user  active               00:00:21.537818  00:00:21.534485   UPDATE task_instance SET last_heartbeat_at='2026-04-23T09:03:18.844164+00:00'::timestamptz, updated_at='2026-04-23T09:03:18.847447+00:00'::timestamptz WHERE task_instance.id
...

after further investigation it seems that the issue we are experiencing is the combination of the idle transactions (i do see idle transactions up to 50 secs durararion as well) and issue described here #63355