Skip to content

Filter mock_ai-related metrics events from telemetry#978

Closed
svarlamov wants to merge 6 commits intomainfrom
devin/1775438423-filter-mock-ai-metrics
Closed

Filter mock_ai-related metrics events from telemetry#978
svarlamov wants to merge 6 commits intomainfrom
devin/1775438423-filter-mock-ai-metrics

Conversation

@svarlamov
Copy link
Copy Markdown
Member

@svarlamov svarlamov commented Apr 6, 2026

Summary

Closes #976. Prevents mock_ai test preset events from reaching telemetry by filtering in observability::log_metrics() — the single lowest-level funnel for all metric events.

A pub(crate) is_mock_ai() helper in src/metrics/mod.rs inspects the serialized MetricEvent and checks two positions:

  1. Attr position 20 (TOOL) — catches AgentUsage / Checkpoint / InstallHooks events where attrs.tool = "mock_ai"
  2. Value position 3 (TOOL_MODEL_PAIRS) — catches Committed events whose tool_model_breakdown keys start with "mock_ai" (e.g. "mock_ai::unknown")

A MOCK_AI_TOOL constant in src/metrics/mod.rs centralizes the string. log_metrics() always runs the filter, then delegates to submit_metric_events() — a no-op in test/test-support builds, real submission otherwise. This means is_mock_ai() is always compiled and exercised by tests with zero cfg annotations on the detection logic itself.

Review & Testing Checklist for Human

  • starts_with(MOCK_AI_TOOL) matching on tool_model_pairs — verify no real tool name could start with "mock_ai" and get falsely filtered.
  • Position constants — the helper uses attr_pos::TOOL (20) and committed_pos::TOOL_MODEL_PAIRS (3). Confirm these match the current sparse array encoding.
  • Integration path not tested in CIsubmit_metric_events is a no-op in test builds, so the filter→submit→telemetry path is never exercised end-to-end in CI. Recommended manual test: run git-ai with the mock_ai preset, make a commit, and verify no events appear in the metrics DB or telemetry upload logs.

Notes

  • Two files touched: src/metrics/mod.rs (MOCK_AI_TOOL constant + is_mock_ai() helper + 5 unit tests) and src/observability/mod.rs (filter call in log_metrics() + submit_metric_events cfg split).
  • All existing + new tests pass; lint and format checks clean.

Link to Devin session: https://app.devin.ai/sessions/9e5cb55142ff40b387b12c28ddf113cb
Requested by: @svarlamov


Open with Devin

Events originating from the mock_ai test preset are now silently
dropped at multiple layers to ensure they never reach telemetry:

- metrics::record() checks EventAttributes::has_mock_ai_tool() and
  returns early for mock_ai events
- record_commit_metrics() checks tool_model_breakdown keys for
  mock_ai prefixes before recording committed events
- MetricEvent::has_mock_ai_tool() inspects serialized attrs and
  committed values for filtering already-stored events
- flush_metrics() in the daemon telemetry worker filters events
  before uploading or storing in the SQLite fallback DB
- flush_metrics_db command discards mock_ai events when draining
  the SQLite queue

Closes #976

Co-Authored-By: Sasha Varlamov <sasha@sashavarlamov.com>
@devin-ai-integration
Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@CLAassistant
Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Copy link
Copy Markdown
Contributor

@devin-ai-integration devin-ai-integration bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 4 additional findings.

Open in Devin Review

devin-ai-integration bot and others added 5 commits April 6, 2026 02:10
… guards

Co-Authored-By: Sasha Varlamov <sasha@sashavarlamov.com>
…elper

Co-Authored-By: Sasha Varlamov <sasha@sashavarlamov.com>
Co-Authored-By: Sasha Varlamov <sasha@sashavarlamov.com>
Co-Authored-By: Sasha Varlamov <sasha@sashavarlamov.com>
Co-Authored-By: Sasha Varlamov <sasha@sashavarlamov.com>
@svarlamov svarlamov closed this Apr 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ensure that mock_ai-related metrics events are filtered out of telemetry

2 participants