Fix scheduler DBDagBag unbounded cache

[jason810496]

• closes: Scheduler DBDagBag cache is never evicted and grows unbounded #69001
• supersedes: Allow DBDagBag TTL cache eviction without a size cap #69774, Use bounded DBDagBag cache in scheduler #69007
• deferred work: Honor the API server Dag cache TTL when no size limit is set #71814, Report Dag cache metrics under each component's own namespace #71815, Make the scheduler Dag cache size and TTL configurable #71816

Why

The scheduler kept every Dag version it deserialized in a mapping that never evicted, so a long-running scheduler grew with the number of versions it had ever seen until it was restarted or OOM killed. Deployments that redeploy Dags frequently accumulate versions fastest and hit this soonest.

How

One line. DBDagBag already builds a bounded LRU cache when given a cache_size; the scheduler simply never passed one.

-        self.scheduler_dag_bag = DBDagBag(load_op_links=False)
+        self.scheduler_dag_bag = DBDagBag(load_op_links=False, cache_size=SCHEDULER_DAG_CACHE_SIZE)

No changes to DBDagBag, to the API server, or to configuration — deliberately, so this cherry-picks cleanly.

Why a size cap rather than a TTL

A least-recently-used cap is the only thing that bounds the cache outright. An idle timeout would not: _get_dag re-checks an entry on each lookup, and cachetools re-arms expiry on assignment, so a TTL reclaims a version only once its runs finish and it stops being requested. That leaves memory a function of the concurrently active set rather than a fixed ceiling.

Where the working set exceeds the cap, the evicted version is re-fetched and deserialized on the next loop. Correctness is unaffected — a cache miss is a re-read, never a wrong Dag.

Known limitation

Enabling the cache also enables DBDagBag's metrics, which are currently hard-coded to api_server.dag_bag.*. Those counters therefore now include scheduler traffic. Giving each component its own namespace requires a DBDagBag API change, deferred to #71815. Making the cap configurable is #71816, and #71814 fixes [api] dag_cache_size = 0 silently ignoring [api] dag_cache_ttl.

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

[ashb]

1024 dag (versions) is probably too aggressive a default?

Do you mean too large or too low by default? Or what threshold do you recommend to set?

I'm thinking its too large. 512 or 256 might be a safer default? (I just worry about how much memory 1024 dags in memory at once could take up

[uranusjr]

Let’s say a dag needs 500 bytes to store a task, and each dag has 100 tasks. That’s 50k for a dag. 256→13m, 512→25m, 1024→50m. It doesn’t sound too bad to me. Let’s do 512?

[uranusjr]

I think we should roll back all API changes in this PR and simply change the scheduler. The scheduler cache also should not rely on any configurations. (It currently still does dag_cache_conf, just to all non-existent configs so only use defaults.) Everything else should be done in later PRs. This is important so the PR can be cherry-picked cleanly.

[jason810496]

I think we should roll back all API changes in this PR and simply change the scheduler. The scheduler cache also should not rely on any configurations. (It currently still does dag_cache_conf, just to all non-existent configs so only use defaults.) Everything else should be done in later PRs. This is important so the PR can be cherry-picked cleanly.

Sure, then I will just make this one a minimal fix (just hardcoded the cache size as 512 for scheduler only) if we don't want to introduce any config at all. Please noted that this means user will need to wait for 3.4.0 to set the cache size via config.

[github-actions]

Backport successfully created: v3-3-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-3-test