Skip to content

fix(telemetry): record development instances only, skip in debug mode - #999

Merged
mikepitre merged 2 commits into
mainfrom
mike/telemetry-dev-only-gate
Oct 1, 2026
Merged

mikepitre merged 2 commits into
mainfrom
mike/telemetry-dev-only-gate

Conversation

@mikepitre

@mikepitre mikepitre commented Oct 1, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

TelemetryCollector.shouldRecord skipped events when the instance type was "development" and recorded everything else. So Android sent telemetry from production apps and dropped it from development apps. clerk-js and clerk-ios do the reverse. The gate came from clerk-ios's guard instanceTypeString() == "development" else { skip }, but the port lost the negation. Its skip reason, "production instance", contradicts the condition it guarded. Mike confirmed the production-only behavior was a bug, not a decision.

The gate now matches clerk-js #shouldRecord. An event passes only when telemetry is enabled, the instance type is "development", and debug mode is off. Sampling and throttling then apply as before. The old code never read isDebugModeEnabled().

ClerkTelemetryEnvironment() reported Clerk.instanceEnvironmentType.name, which is "DEVELOPMENT". It now reports the enum's serial name, "development", which is the value clerk-js sends in the it field.

Behavior change. Production apps (pk_live_ keys) stop sending telemetry. Development apps (pk_test_ keys) with telemetry enabled and debug mode off start sending it. With enableDebugMode = true the SDK sends nothing, which matches clerk-js. clerk-ios still records in debug mode and only adds logging, so on this point Android follows clerk-js rather than clerk-ios.

The flush() hunk only changes formatting. Spotless ratchets from origin/main and reformats every touched file with ktfmt 0.64.

Testing

  • The new TelemetryCollectorTest runs the real TelemetryCollector against a Ktor MockEngine and asserts which it values reach /v1/event. It covers development (sent as "development"), production, unknown, telemetry disabled, and debug mode. Two more cases use the real ClerkTelemetryEnvironment() with a pk_test_ key and a pk_live_ key. On main, 5 of the 7 tests fail. The live key gives expected:<[]> but was:<[PRODUCTION]>, and development gives expected:<[development]> but was:<[]>. With the fix, all 7 pass.
  • The debug-mode test also passes on main, but only because the inverted gate already drops development events. I applied the fix, deleted the debug branch, and skipsWhenDebugModeIsEnabled failed. So the test does guard that branch.
  • ./gradlew :source:telemetry:testAndroidHostTest :source:telemetry:spotlessCheck :source:telemetry:detekt passes (9 tests, 0 failures). With this branch merged with fix(ui): share one process-wide TelemetryCollector #1000, :source:ui:testDebugUnitTest also passes on JDK 21 (506 tests, 0 failures).
  • This PR adds io.ktor:ktor-client-mock:3.5.2 to the version catalog. Only androidHostTest uses it.

Overlap with open PRs

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes
    • Telemetry events are now recorded for development instances, including those using test publishable keys. Events continue to be skipped for production or unknown instances, live publishable keys, and when telemetry is disabled or debug mode is enabled.
  • Tests
    • Added coverage for telemetry behavior across development, production, unknown, and test-key environments.

mikepitre and others added 2 commits October 1, 2026 10:24
Drives TelemetryCollector against a Ktor MockEngine and asserts which
instance types reach the wire. Fails on main: development events are
dropped and production events are sent with it="PRODUCTION".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The gate was ported from clerk-ios' `guard instanceType == "development"
else { skip }` without the negation, so Android sent telemetry from
production instances and dropped it from development ones. Match
clerk-js: record only when the instance type is "development" and debug
mode is off.

ClerkTelemetryEnvironment reported the enum constant name
("DEVELOPMENT"). Report the serial name ("development") so the `it`
field matches clerk-js and clerk-ios.

The flush() hunk is ktfmt 0.64 reformatting from spotless ratcheting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: clerk/clerk-android/.coderabbit.yaml

Review profile: CHILL

Plan: Team

Run ID: c1f40b6f-8160-46ca-8da2-ead7b603c452

📥 Commits

Reviewing files that changed from the base of the PR and between c58b14e and ba0945e.

📒 Files selected for processing (5)
  • gradle/libs.versions.toml
  • source/telemetry/build.gradle.kts
  • source/telemetry/src/androidHostTest/kotlin/com/clerk/telemetry/TelemetryCollectorTest.kt
  • source/telemetry/src/androidMain/kotlin/com/clerk/telemetry/ClerkTelemetryEnvironment.kt
  • source/telemetry/src/commonMain/kotlin/com/clerk/telemetry/TelemetryCollector.kt

Limit details: You’ve used all 4 included reviews currently available. Your 61 included PR review attempts over the past 7 days set your current allowance at 4 reviews per hour.


📝 Walkthrough

Walkthrough

The telemetry environment now supplies a serialized instance type. The collector records events only for development instances when debug mode is off. New host tests check these conditions and use a mock HTTP client to inspect flushed events.

Changes

Telemetry filtering

Layer / File(s) Summary
Instance type serialization and event filtering
source/telemetry/src/androidMain/..., source/telemetry/src/commonMain/...
The environment returns the serialized name for its instance type. The collector rejects non-development instances and events when debug mode is enabled.
Mocked telemetry tests
gradle/libs.versions.toml, source/telemetry/build.gradle.kts, source/telemetry/src/androidHostTest/...
The host test source set adds the Ktor mock client. Tests check event sending for development instances and test keys, and event suppression for production or unknown instances, disabled telemetry, debug mode, and live keys.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~12 minutes

Change: Bug fix

Suggested reviewers: swolfand

Merge Risk: ⚪ Minimal · up to ba094

This change fixes telemetry gating so that only development instances with telemetry enabled and debug mode off send events. No actionable merge-blocking risk was identified.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 14 functions across 4 files. (1 skipped: 1… Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly and concisely describes the main telemetry behavior change: recording development instances only and skipping debug mode.
Description check ✅ Passed The description is detailed and on topic. It explains the bug, intended behavior, implementation changes, testing, and related PR overlap. It uses custom headings instead of the template headings, but…
Full details: Docstring Coverage

Explanation

Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 14 functions across 4 files. (1 skipped: 1 unsupported.)

  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.


Comment @coderabbitai help to get the list of available commands.

@mikepitre

Copy link
Copy Markdown
Collaborator Author

Verified on an Android emulator

Setup. Workbench app on a clean Pixel (API 36) emulator, development key for amusing-barnacle-26. For each build I did a clean install, captured all emulator traffic with adb emu network capture, and opened the prebuilt AuthView 31 times. viewDidAppear samples at 10%, so 31 appearances give about a 96% chance of at least one recorded event. Then I waited 40 seconds for the 30-second flush. To exercise the gate, I patched workbench locally (not committed) so enableDebugMode = false.

Build Debug mode Traffic to clerk-telemetry.com
main (c58b14e) off none (0 DNS or TLS hits)
this PR (ba0945e) off yes (DNS query and TLS SNI for clerk-telemetry.com)
this PR (ba0945e) on none

main never sends anything for a development key, because its gate skips development instances. With this PR, development instances send events, and debug mode suppresses them, matching clerk-js collector.ts. The capture shows that a connection was made, not the event payload, which is encrypted. The payload shape is covered by TelemetryCollectorTest. Production keys aren't covered here because the test instance is development-only.

@swolfand swolfand left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The gate logic and the lowercase environment name are correct. Non-blocking: CI doesn't run :source:telemetry tests, so the new TelemetryCollectorTest only runs locally. Worth adding it to the test matrix in a follow-up. The check against "development" is now an exact match, where the old one ignored case. That's fine for the built-in environment, but it matters for any custom TelemetryEnvironment. LGTM.

@mikepitre
mikepitre merged commit 05411b0 into main Oct 1, 2026
9 checks passed
@mikepitre
mikepitre deleted the mike/telemetry-dev-only-gate branch October 1, 2026 19:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants