Add telemetry instrumentation for azd tool first-run and operations#8261
Open
hemarina wants to merge 4 commits into
Open
Add telemetry instrumentation for azd tool first-run and operations#8261hemarina wants to merge 4 commits into
hemarina wants to merge 4 commits into
Conversation
hemarina
changed the title
…na/tool-telemetry-8168
hemarina
requested review from
JeffreyCA,
jongio,
rajeshkamal5050,
tg-msft,
vhvb1989 and
weikanglim
as code owners
Contributor
There was a problem hiding this comment.
Pull request overview
Adds OpenTelemetry usage-attribute instrumentation for the azd tool command group and its first-run onboarding middleware, enabling measurement of first-run funnel behavior and tool operation outcomes while avoiding raw error strings in tool-specific telemetry.
Changes:
- Introduces new
tool.*telemetry field constants and documents the emitted schema. - Instruments
tool install/upgrade/check/showand the tool first-run middleware to emit usage attributes on the command span. - Extends telemetry coverage tests and adds unit tests for new tool telemetry helpers/paths.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| cli/azd/internal/tracing/fields/fields.go | Adds new tool.* attribute key constants for first-run and tool operations telemetry. |
| cli/azd/docs/tracing-in-azd.md | Documents the new tool telemetry schema and coverage expectations. |
| cli/azd/cmd/tool.go | Emits tool operation telemetry attributes and adds helpers to aggregate install/upgrade results. |
| cli/azd/cmd/tool_test.go | Adds unit tests validating aggregate tool install/upgrade telemetry emission behavior. |
| cli/azd/cmd/telemetry_coverage_test.go | Registers tool subcommands for telemetry coverage and asserts new field constants. |
| cli/azd/cmd/middleware/tool_first_run.go | Emits first-run funnel telemetry attributes (skip reason, opt-in, selection, install outcomes). |
| cli/azd/cmd/middleware/tool_first_run_test.go | Adds tests validating first-run skip-reason emission and silent skip paths. |
| cli/azd/.vscode/cspell.yaml | Adds spelling dictionary entries for newly introduced telemetry terms. |
Comments suppressed due to low confidence (1)
cli/azd/cmd/tool.go:1063
tool.idis set from the raw CLI arg beforeFindToolvalidates it. If the arg is invalid, this can emit arbitrary user input into telemetry and inflate cardinality. Move theSetUsageAttributes(fields.ToolIdKey...)call to afterFindToolsucceeds (or set it fromtoolDef.Id).
toolID := a.args[0]
tracing.SetUsageAttributes(fields.ToolIdKey.String(toolID))
toolDef, err := a.manager.FindTool(toolID)
if err != nil {
return nil, fmt.Errorf("finding tool: %w", err)
Comment on lines
+484
to
+487
| tracing.SetUsageAttributes( | ||
| fields.ToolIdsKey.String(strings.Join(ids, ",")), | ||
| fields.ToolDryRunKey.Bool(a.flags.dryRun), | ||
| ) |
Comment on lines
+343
to
+347
| tracing.SetUsageAttributes( | ||
| fields.ToolFirstRunToolsSelectedKey.Int(len(selectedIDs)), | ||
| fields.ToolFirstRunToolsSelectedNamesKey.String(strings.Join(selectedIDs, ",")), | ||
| fields.ToolFirstRunToolsDeselectedNamesKey.String(strings.Join(deselectedIDs, ",")), | ||
| ) |
Comment on lines
+479
to
+486
| // ToolFirstRunCompletedKey records whether the first-run experience | ||
| // reached the point where `tool.firstRunCompleted` was persisted to user | ||
| // config. False when the user cancelled or detection failed. | ||
| ToolFirstRunCompletedKey = AttributeKey{ | ||
| Key: attribute.Key("tool.firstrun.completed"), | ||
| Classification: SystemMetadata, | ||
| Purpose: FeatureInsight, | ||
| } |
Comment on lines
+216
to
+229
| | Attribute | Type | Emitted when | Notes | | ||
| | --- | --- | --- | --- | | ||
| | `tool.firstrun.skip_reason` | string | First-run was bypassed | One of `env_var`, `no_prompt`, `ci_cd`, `non_interactive`, `already_completed`, `config_error`. The alpha-disabled and child-action skip paths are intentionally silent because the user has no opportunity to opt in. | | ||
| | `tool.firstrun.optin` | bool | User answered the welcome prompt | `true` = accepted, `false` = declined. | | ||
| | `tool.firstrun.tools_detected` | int | Optin = `true` | Count of built-in tools already installed locally. | | ||
| | `tool.firstrun.tools_offered` | int | Optin = `true` | Count of recommended tools offered for installation. `0` when nothing is missing. | | ||
| | `tool.firstrun.tools_selected` | int | At least one tool was offered | Count of tools the user kept selected. | | ||
| | `tool.firstrun.tools_selected_names` | string | At least one tool selected | Comma-separated built-in tool IDs (low-cardinality, no PII). | | ||
| | `tool.firstrun.tools_deselected_names` | string | User deselected at least one offered tool | Comma-separated built-in tool IDs. | | ||
| | `tool.firstrun.completed` | bool | First-run flow reached the completion-persistence step | `true` once `tool.firstRunCompleted` is written to user config. | | ||
| | `tool.firstrun.install_success_count` | int | First-run installed tools | Number of tools that succeeded during the first-run batch install. Mirrors `tool.install.success_count` but is namespaced so a subsequent `azd tool install` action on the same span does not overwrite it. | | ||
| | `tool.firstrun.install_failure_count` | int | First-run installed tools | Number of tools that failed during the first-run batch install. | | ||
| | `tool.firstrun.install_failed_ids` | string | First-run had at least one failure | Comma-separated tool IDs whose first-run install failed. | | ||
| | `tool.firstrun.install_duration_ms` | int | First-run installed tools | Wall-clock duration of the first-run install batch in milliseconds. | |
Comment on lines
+235
to
+248
| | Attribute | Type | Emitted by | Notes | | ||
| | --- | --- | --- | --- | | ||
| | `tool.id` | string | Single-target install / upgrade / show | The built-in tool identifier. | | ||
| | `tool.ids` | string | Batch install / upgrade | Comma-separated tool IDs. | | ||
| | `tool.dry_run` | bool | install / upgrade | Reflects the `--dry-run` flag. | | ||
| | `tool.install.strategy` | string | Single-target install / upgrade | E.g. `winget`, `brew`, `manual`. | | ||
| | `tool.install.success` | bool | Single-target install / upgrade | Whether the per-tool operation succeeded. | | ||
| | `tool.install.success_count` | int | Batch install / upgrade | Number of tools that succeeded. | | ||
| | `tool.install.failure_count` | int | Batch install / upgrade | Number of tools that failed. | | ||
| | `tool.install.failed_ids` | string | At least one failure | Comma-separated tool IDs whose operation failed. **Only tool IDs are recorded — error messages flow through the global error middleware (`error.message`).** | | ||
| | `tool.install.duration_ms` | int | Batch install / upgrade | Wall-clock duration of the operation in milliseconds. | | ||
| | `tool.upgrade.from_version` | string | Single-target upgrade | Pre-upgrade installed version (when detection was run). | | ||
| | `tool.upgrade.to_version` | string | Single-target upgrade | Post-upgrade installed version. | | ||
| | `tool.check.updates_available` | int | `tool check` | Count of tools whose `UpdateAvailable` is `true`. | |
Collaborator
Azure Dev CLI Install InstructionsInstall scriptsMacOS/Linux
bash: pwsh: WindowsPowerShell install MSI install Standalone Binary
MSI
Documentationlearn.microsoft.com documentationtitle: Azure Developer CLI reference
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #8168.
Summary
Instruments the
azd toolfeature so we can answer the questions in the issue: adoption of the first-run experience, tool preferences, skip rate, install success rate, and command engagement outside first-run.All attributes are attached as usage attributes via
tracing.SetUsageAttributes, so they land on the user''s actual command span (e.g.cmd.tool.install) rather than on a separate child span — matching existing azd telemetry conventions.What''s emitted
First-run experience (
cmd/middleware/tool_first_run.go)tool.firstrun.skip_reasonenv_var,no_prompt,ci_cd,non_interactive,already_completed,config_error)tool.firstrun.optintrue= accepted)tool.firstrun.tools_detectedtool.firstrun.tools_offeredtool.firstrun.tools_selectedtool.firstrun.tools_selected_namestool.firstrun.tools_deselected_namestool.firstrun.completedtool.firstRunCompletedwas persisted to user configtool.firstrun.install_success_counttool.firstrun.install_failure_counttool.firstrun.install_failed_idstool.firstrun.install_duration_msTool commands (
cmd/tool.go—install,upgrade,check,show)tool.idtool.idstool.dry_runinstall/upgradewith--dry-runtool.install.strategytool.install.successtool.install.success_count/tool.install.failure_counttool.install.failed_idstool.install.duration_mstool.upgrade.from_version/tool.upgrade.to_versiontool.check.updates_availabletool checkDesign notes / deviations from the issue''s proposed schema
The "Proposed Telemetry Events" table in #8168 was a sketch; the implementation refines a few things:
tool.id/tool.idsinstead of separatetool.install.tool_idandtool.upgrade.tool_id. The command name already disambiguates; one set of keys keeps the schema small.tool.install.countintosuccess_count+failure_count(+failed_ids). Lets backends compute success rate directly without joining events.tool.install.errorstrings in favor oftool.install.failed_ids. Error strings risk leaking paths, usernames, or proxy URLs; the global error middleware still captures error class on the same span.tool.firstrun.optinbool instead of two separateoptin_accepted/optin_declinedattributes. Same signal, half the schema.tool.firstrun.install_*so that when a user''s very firstazdcommand isazd tool install <x>, the first-run signal is not clobbered by the command-leveltool.install.*write on the same span (usage attributes are last-write-wins).tool.dry_runsince bothinstallandupgradeaccept--dry-runand the flag materially changes what the command did.Privacy
Only built-in tool IDs (e.g.
az-cli,docker) and semver version strings are captured. No file paths, no user-identifiable data, no raw error text.Tests
cmd/middleware/tool_first_run_test.go—TestToolFirstRunMiddleware_EmitsSkipReason(env-var / no-prompt / ci_cd / non-interactive / already-completed) +TestToolFirstRunMiddleware_NoSkipReasonForSilentPaths.cmd/tool_test.go— 4 cases foremitToolInstallTelemetry(all-success batch, all-failure batch, mixed, empty result set).cmd/telemetry_coverage_test.go—tool,tool list,tool check,tool install,tool show,tool upgraderegistered; newToolFieldssubtest asserts every newAttributeKeyresolves to its expectedtool.*string.Docs
docs/tracing-in-azd.md— new "Tool Command Telemetry" section.Out of scope
pkg/tool/installer.go,pkg/tool/detector.go,pkg/tool/update_checker.go— instrumentation lives at the command/middleware layer (where decisions are made) rather than inside the orchestration types. That matches how other azd domains are instrumented.