Filter mock_ai-related metrics events from telemetry#978
Closed
Filter mock_ai-related metrics events from telemetry#978
Conversation
Events originating from the mock_ai test preset are now silently dropped at multiple layers to ensure they never reach telemetry: - metrics::record() checks EventAttributes::has_mock_ai_tool() and returns early for mock_ai events - record_commit_metrics() checks tool_model_breakdown keys for mock_ai prefixes before recording committed events - MetricEvent::has_mock_ai_tool() inspects serialized attrs and committed values for filtering already-stored events - flush_metrics() in the daemon telemetry worker filters events before uploading or storing in the SQLite fallback DB - flush_metrics_db command discards mock_ai events when draining the SQLite queue Closes #976 Co-Authored-By: Sasha Varlamov <sasha@sashavarlamov.com>
Contributor
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
|
|
… guards Co-Authored-By: Sasha Varlamov <sasha@sashavarlamov.com>
…elper Co-Authored-By: Sasha Varlamov <sasha@sashavarlamov.com>
Co-Authored-By: Sasha Varlamov <sasha@sashavarlamov.com>
Co-Authored-By: Sasha Varlamov <sasha@sashavarlamov.com>
Co-Authored-By: Sasha Varlamov <sasha@sashavarlamov.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #976. Prevents
mock_aitest preset events from reaching telemetry by filtering inobservability::log_metrics()— the single lowest-level funnel for all metric events.A
pub(crate) is_mock_ai()helper insrc/metrics/mod.rsinspects the serializedMetricEventand checks two positions:TOOL) — catches AgentUsage / Checkpoint / InstallHooks events whereattrs.tool = "mock_ai"TOOL_MODEL_PAIRS) — catches Committed events whosetool_model_breakdownkeys start with"mock_ai"(e.g."mock_ai::unknown")A
MOCK_AI_TOOLconstant insrc/metrics/mod.rscentralizes the string.log_metrics()always runs the filter, then delegates tosubmit_metric_events()— a no-op in test/test-support builds, real submission otherwise. This meansis_mock_ai()is always compiled and exercised by tests with zerocfgannotations on the detection logic itself.Review & Testing Checklist for Human
starts_with(MOCK_AI_TOOL)matching ontool_model_pairs— verify no real tool name could start with"mock_ai"and get falsely filtered.attr_pos::TOOL(20) andcommitted_pos::TOOL_MODEL_PAIRS(3). Confirm these match the current sparse array encoding.submit_metric_eventsis a no-op in test builds, so the filter→submit→telemetry path is never exercised end-to-end in CI. Recommended manual test: rungit-aiwith themock_aipreset, make a commit, and verify no events appear in the metrics DB or telemetry upload logs.Notes
src/metrics/mod.rs(MOCK_AI_TOOLconstant +is_mock_ai()helper + 5 unit tests) andsrc/observability/mod.rs(filter call inlog_metrics()+submit_metric_eventscfg split).Link to Devin session: https://app.devin.ai/sessions/9e5cb55142ff40b387b12c28ddf113cb
Requested by: @svarlamov