[v3-3-test] Report Dag cache metrics under each component's own namespace (#71815) - #71925
Open
jason810496 wants to merge 1 commit into
Open
[v3-3-test] Report Dag cache metrics under each component's own namespace (#71815)#71925jason810496 wants to merge 1 commit into
jason810496 wants to merge 1 commit into
Conversation
…pace (apache#71815) * Report Dag cache metrics under each component's own namespace Every DBDagBag emitted its cache counters under api_server.dag_bag.*, so once the scheduler gained a cache its traffic was silently counted against the API server's series. An operator reading cache_hit or cache_size could not tell the two components apart, and the API server's numbers became wrong rather than merely incomplete. Each caller now supplies the namespace it reports under, and a cache built without one fails at construction instead of emitting a partially-formed metric name mid-request. The registry check matched dynamic metric names only by the static prefix ahead of their first variable, which cannot express a name assembled from a caller supplied prefix. It now matches on all static parts wherever the variable sits. * Scope Dag cache configuration to cached database Dag bags DBDagBag is used by callers that do not configure cache eviction or metrics. Keeping those concerns behind a dedicated subtype prevents optional constructor combinations and makes metric ownership explicit. * Preserve no-eviction Dag cache configuration * Avoid redundant Dag cache checks (cherry picked from commit bcf1dbf) Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
jason810496
requested review from
XD-DENG,
amoghrajesh,
ashb,
bugraoz93,
choo121600,
ephraimbuddy,
gopidesupavan,
henry3260,
jscheffl,
pierrejeambrun,
potiuk,
rawwar and
shubhamraj-git
as code owners
August 21, 2026 07:27
boring-cyborg
Bot
added
area:API
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Every configured
DBDagBagemitted cache counters underapi_server.dag_bag.*. Once the scheduler gained a bounded cache in #71704, scheduler traffic was counted against the API server's series, so operators could not distinguish the two components.The cache settings and metrics also made every
DBDagBagcaller opt into an interface intended only for long-lived component caches.How
DBDagBagremains the simple base implementation for callers that do not configure eviction or cache metrics.CachedDBDagBagowns LRU, TTL, and no-eviction cache modes, along with thread safety and the required component metric namespace.The API server always constructs
CachedDBDagBagfrom its cache configuration. With size and TTL both set to 0, the subclass uses a plain unbounded dict and performs no eviction. The scheduler constructsCachedDBDagBagunderscheduler.dag_bag.*.The metrics registry check now matches every static part of a dynamic metric name, including names assembled from a caller-supplied prefix.
What
api_server.dag_bag.*.scheduler.dag_bag.*.DBDagBag.0/0no-eviction behavior while keeping configured component caches onCachedDBDagBag.cherry picked from commit bcf1dbf)