Skip to content

Apply share_labels/target_info tags regardless of payload order - #24884

Open
ian28223 wants to merge 2 commits into
masterfrom
ian.bucad/fix_openmetrics_share_labels_ordering
Open

Apply share_labels/target_info tags regardless of payload order#24884
ian28223 wants to merge 2 commits into
masterfrom
ian.bucad/fix_openmetrics_share_labels_ordering

Conversation

@ian28223

@ian28223 ian28223 commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes LabelAggregator (OpenMetrics V2) so share_labels/target_info tags are collected from the full scrape payload before any metric is yielded, instead of interleaving collection with yielding.

Motivation

  • If the shared-label source metric appears after a counter in the payload, that counter is yielded untagged on the first scrape, then tagged on every scrape after.
  • For a monotonic counter, that tag change looks like a brand new context, and combined with the scraper's flush_first_value flag already being True by the second scrape, its full raw counter value gets flushed as a single interval's delta instead of a small diff.
  • Root-caused via a RabbitMQ customer escalation (rabbitmq.queue.messages.acked.count spiking to the counter's entire lifetime total on Agent restart); reproduces identically on the generic openmetrics check, confirming it's a base-check issue.

Review checklist (to be filled by reviewers)

  • Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
  • Add qa/required if this PR needs QA validation, or qa/skip-qa if it does not. Exactly one of the two is required.
  • If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

ian28223 added a commit that referenced this pull request Aug 17, 2026
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@ian28223 ian28223 added the qa/required QA is required for this PR and will generate a QA card label Aug 17, 2026
@datadog-prod-us1-5

datadog-prod-us1-5 Bot commented Aug 17, 2026

Copy link
Copy Markdown

Tests  Code Coverage

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 88.63% (+0.10%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: fedf0af | Docs | Datadog PR Page | Give us feedback!

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Aug 17, 2026

Copy link
Copy Markdown

evalya-impact-summary

evalya impact analysis
Impact analysis: 0 selected, 0 skipped (of 0 test tasks)
Publish tasks:   1 (always emitted)
Diff (4 files):
  datadog_checks_base/changelog.d/24884.fixed
  datadog_checks_base/datadog_checks/base/checks/openmetrics/v2/labels.py
  datadog_checks_base/tests/base/checks/openmetrics/test_v2/scraper/test_http_status_class_scraper.py
  datadog_checks_base/tests/base/checks/openmetrics/test_v2/test_options.py

Debug a specific task: evalya plan impact --path <path> --task <task>

Learn more about CI impact filtering

ian28223 added a commit that referenced this pull request Aug 17, 2026
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@ian28223
ian28223 force-pushed the ian.bucad/fix_openmetrics_share_labels_ordering branch from 79836cf to 3a0e577 Compare August 17, 2026 09:07
@ian28223
ian28223 marked this pull request as ready for review August 17, 2026 09:07
@ian28223
ian28223 requested review from a team as code owners August 17, 2026 09:07

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3a0e577f88

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread datadog_checks_base/datadog_checks/base/checks/openmetrics/v2/labels.py Outdated
ian28223 and others added 2 commits August 17, 2026 21:55
…d order

LabelAggregator collected and yielded metrics in the same interleaved pass:
a metric processed before its shared-label source metric was reached in the
payload would be yielded without the shared tags, while the same metric on
a later scrape (once shared_labels_cached is True) would have them applied.
For a monotonic counter, that tag-set change looks like a brand new context
to the aggregator, and combined with the scraper-wide flush_first_value flag
already being true by the second scrape, its full raw counter value gets
flushed as if it were a single interval's delta instead of a small diff.

Buffer metrics only until every configured shared-label source has been
found, then yield the buffered prefix and continue streaming the rest, so
tagging no longer depends on payload order without buffering the entire
response into memory.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@ian28223
ian28223 force-pushed the ian.bucad/fix_openmetrics_share_labels_ordering branch from 3a0e577 to fedf0af Compare August 17, 2026 11:56
@dd-octo-sts

dd-octo-sts Bot commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Validation Report

All 21 validations passed.

Show details
Validation Description Status
agent-reqs Verify check versions match the Agent requirements file
ci Validate CI configuration and code coverage settings
codeowners Validate every integration has a CODEOWNERS entry
config Validate default configuration files against spec.yaml
dep Verify dependency pins are consistent and Agent-compatible
http Validate integrations use the HTTP wrapper correctly
imports Validate check imports do not use deprecated modules
integration-style Validate check code style conventions
jmx-metrics Validate JMX metrics definition files and config
labeler Validate PR labeler config matches integration directories
legacy-signature Validate no integration uses the legacy Agent check signature
license-headers Validate Python files have proper license headers
licenses Validate third-party license attribution list
metadata Validate metadata.csv metric definitions
models Validate configuration data models match spec.yaml
openmetrics Validate OpenMetrics integrations disable the metric limit
package Validate Python package metadata and naming
qa-label Validate the pull request declares whether it needs QA for the next Agent release
readmes Validate README files have required sections
saved-views Validate saved view JSON file structure and fields
version Validate version consistency between package and changelog

View full run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant