Skip to content

Honor the API server Dag cache TTL when no size limit is set - #71814

Merged
jason810496 merged 5 commits into
apache:mainfrom
jason810496:fix/api-dag-cache-ttl-ignored
Aug 19, 2026
Merged

Honor the API server Dag cache TTL when no size limit is set#71814
jason810496 merged 5 commits into
apache:mainfrom
jason810496:fix/api-dag-cache-ttl-ignored

Conversation

@jason810496

@jason810496 jason810496 commented Aug 19, 2026

Copy link
Copy Markdown
Member

[api] dag_cache_size = 0 reads as "no size limit", but it selected a mapping that never evicted and silently ignored [api] dag_cache_ttl. Age-based eviction could only be enabled by also accepting a size cap.

A TTL now applies with or without a size limit; only setting both options to 0 disables eviction. Deployments on dag_cache_size = 0 start evicting on the default hour-long TTL — set dag_cache_ttl = 0 to keep the old behaviour.

Also corrects the documented version_added for both options: 3.2.2, not 3.3.0.


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

@uranusjr

Copy link
Copy Markdown
Member

We should probably add a news fragment to the behavioral change when dag_cache_size = 0 since it’s visible to the user. The change is justified as a bug fix, but the fix may in turn cause issues in existing deployments.

Comment thread airflow-core/docs/administration-and-deployment/web-stack.rst Outdated
Comment thread airflow-core/src/airflow/models/dagbag.py Outdated

@uranusjr uranusjr left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few nits.

@jason810496
jason810496 marked this pull request as draft August 19, 2026 09:03
`[api] dag_cache_size = 0` reads as "no size limit", but it selected a mapping
that never evicted at all and silently ignored `[api] dag_cache_ttl`. Age-based
eviction could therefore only be enabled by also accepting a size cap, which is
backwards for a deployment that wants to bound staleness rather than count.

A TTL now applies with or without a size limit, and only setting both options to
0 disables eviction. Deployments on `dag_cache_size = 0` start evicting on the
default hour-long TTL; setting `dag_cache_ttl = 0` restores the old behaviour.

The options were also documented as added in 3.3.0. They shipped in 3.2.2.
Negative cache configuration should fail fast, and operator guidance must reflect that TTL refreshes happen only at revalidation boundaries.
@jason810496
jason810496 force-pushed the fix/api-dag-cache-ttl-ignored branch from a28d57e to 5f6c258 Compare August 19, 2026 09:47
@jason810496
jason810496 requested a lite review from Copilot August 19, 2026 09:48

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Validating at the shared boundary prevents non-API callers from silently clamping invalid cache settings.
Context-specific errors help operators identify invalid API settings, while defensive constructor checks protect other callers and documentation remains compatible with the spellchecker.
@jason810496
jason810496 marked this pull request as ready for review August 19, 2026 12:59

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably add a news fragment to the behavioral change when dag_cache_size = 0 since it’s visible to the user. The change is justified as a bug fix, but the fix may in turn cause issues in existing deployments.

Thanks TP for the review. I’ve resolved all the comments, and it makes sense to add the new fragment.

Comment thread airflow-core/docs/administration-and-deployment/web-stack.rst Outdated
Comment thread airflow-core/src/airflow/models/dagbag.py Outdated
@jason810496
jason810496 merged commit f63d321 into apache:main Aug 19, 2026
140 of 159 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:API Airflow's REST/HTTP API area:ConfigTemplates area:Scheduler including HA (high availability) scheduler kind:documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants