Add governed OTel identity capture for Local agents - #336814
Draft
Ross Wollman (rwoll) wants to merge 3 commits into
Draft
Ross Wollman (rwoll) wants to merge 3 commits into
Ross Wollman (rwoll) wants to merge 3 commits into
Conversation
Keep account and resource identity default-off, enforce explicit managed denial at all export boundaries, and preserve late-policy recovery behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Preserve empty and future-only telemetry block presence through normalization and serialization. Keep unrelated policy keys and sandbox precedence unchanged, and observe native content controls only for source selection. Add normalization, multi-source, transport, and policy-update regressions. Core execution and follow-up lint remain blocked in the local validation environment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Identity can be re-enabled without reload, and tool descriptions can leak while content capture is disabled.
Get a fresh assessment by requesting another Copilot review.
Review effort: Balanced
Findings: 2
Open (3)
What changed in this PR
Adds independently governed identity capture to Local Copilot OTel while preserving managed-policy precedence and immediate revocation.
Changes:
- Adds identity policy projection and whole-block telemetry source selection.
- Filters identity across OTel spans, resources, logs, metrics, SQLite, and local events.
- Adds identity attribution, documentation, and regression coverage.
| File | Description |
|---|---|
src/vs/workbench/services/policies/test/browser/accountPolicyService.test.ts |
Tests telemetry source replacement. |
src/vs/workbench/services/accounts/test/browser/managedSettings.test.ts |
Tests server adaptation. |
src/vs/workbench/services/accounts/browser/managedSettings.ts |
Types identity capture responses. |
src/vs/platform/policy/test/node/nativeManagedSettingsService.test.ts |
Tests native capture controls. |
src/vs/platform/policy/test/common/fileManagedSettingsService.test.ts |
Tests telemetry block preservation. |
src/vs/platform/policy/test/common/copilotManagedSettings.test.ts |
Tests projection and precedence. |
src/vs/platform/policy/common/copilotManagedSettings.ts |
Implements policy selection and normalization. |
src/vs/platform/configuration/test/common/policyConfiguration.test.ts |
Tests nullable policy references. |
src/vs/platform/agentHost/OTEL.md |
Documents policy ownership. |
src/vs/platform/agentHost/common/agentHostStarter.config.contribution.ts |
Registers the shared policy. |
extensions/copilot/src/platform/otel/node/test/otelIdentity.spec.ts |
Tests identity filtering. |
extensions/copilot/src/platform/otel/node/otelServiceImpl.ts |
Enforces identity at OTel boundaries. |
extensions/copilot/src/platform/otel/node/otelIdentityExporters.ts |
Adds filtering exporters. |
extensions/copilot/src/platform/otel/node/inMemoryOTelService.ts |
Filters local event delivery. |
extensions/copilot/src/platform/otel/common/test/otelConfigResolution.spec.ts |
Tests identity resolution. |
extensions/copilot/src/platform/otel/common/test/agentOTelEnv.spec.ts |
Tests environment isolation. |
extensions/copilot/src/platform/otel/common/otelIdentity.ts |
Adds identity helpers. |
extensions/copilot/src/platform/otel/common/otelConfigResolution.ts |
Resolves policy-aware settings. |
extensions/copilot/src/platform/otel/common/otelConfig.ts |
Adds identity configuration. |
extensions/copilot/src/platform/otel/common/genAiAttributes.ts |
Defines identity attributes. |
extensions/copilot/src/platform/otel/common/agentOTelEnv.ts |
Documents environment boundaries. |
extensions/copilot/src/platform/configuration/common/configurationService.ts |
Registers the nullable setting key. |
extensions/copilot/src/extension/prompt/node/searchSubagentToolCallingLoop.ts |
Supplies authentication state. |
extensions/copilot/src/extension/prompt/node/executionSubagentToolCallingLoop.ts |
Supplies authentication state. |
extensions/copilot/src/extension/prompt/node/defaultIntentRequestHandler.ts |
Supplies authentication state. |
extensions/copilot/src/extension/prompt/node/codebaseToolCalling.ts |
Supplies authentication state. |
extensions/copilot/src/extension/otel/common/test/otelStaleConfigMonitor.spec.ts |
Tests revocation drift. |
extensions/copilot/src/extension/mcp/vscode-node/mcpToolCallingLoop.tsx |
Supplies authentication state. |
extensions/copilot/src/extension/intents/test/node/toolCallingLoopIdentity.spec.ts |
Tests invocation attribution. |
extensions/copilot/src/extension/intents/node/toolCallingLoop.ts |
Attributes agent spans. |
extensions/copilot/src/extension/inlineChat2/node/inlineChatIntent.ts |
Attributes inline spans. |
extensions/copilot/src/extension/extension/vscode-node/services.ts |
Adds live permission checks. |
extensions/copilot/package.json |
Contributes the identity setting. |
extensions/copilot/docs/monitoring/agent_monitoring.md |
Documents governed capture. |
build/lib/policies/policyData.jsonc |
Exports policy metadata. |
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+311
to
+312
| builder.define(IOTelService, new NodeOTelService(otelConfig, logFn, otelConfig.dbSpanExporter ? otelSqliteStore : undefined, | ||
| () => otelConfigResolver.resolve().config.captureIdentity)); |
Comment on lines
+18
to
+20
| GenAiAttr.INPUT_MESSAGES, GenAiAttr.OUTPUT_MESSAGES, GenAiAttr.SYSTEM_INSTRUCTIONS, | ||
| GenAiAttr.TOOL_DEFINITIONS, GenAiAttr.TOOL_CALL_ARGUMENTS, GenAiAttr.TOOL_CALL_RESULT, | ||
| CopilotChatAttr.USER_REQUEST, CopilotChatAttr.REASONING_CONTENT, CopilotChatAttr.PROMPT_CONTEXT, |
| settingProtocol: read<string>('protocol') || undefined, | ||
| settingServiceName: read<string>('serviceName') || undefined, | ||
| settingResourceAttributes: read<Record<string, string>>('resourceAttributes'), | ||
| policyResourceAttributes: hasEnterpriseSettings ? read<Record<string, string>>('resourceAttributes') : undefined, |
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #302930.
Summary
Adds opt-in, independently governed OTel identity capture to the Local harness. This PR now targets
main; the managed OTel startup-recovery prerequisite, #336701, has merged.telemetry.capture.identitycontrols authenticated account attribution (user.nameon agent spans), OS account (process.user.name), and machine name (host.name). Capture defaults off and does not enable prompt, response, or tool-content capture.falseoverrides personal/environment opt-in. The extension keeps omission distinct from denial, and rechecks permission at export/event boundaries so revocation does not wait for a reload.Native Agent Host identity enforcement remains runtime-owned. This does not implement the other five granular content-capture controls, resource-attribute placeholder expansion, or subagent cost accounting. External span content filtering respects the existing
captureContentswitch; local debug content behavior is unchanged.Compatibility: An identity-only managed block also replaces personal exporter settings with policy/schema defaults. Configure managed enablement and destination when export should continue. Explicit identity resource attributes require identity permission; they do not bypass default-off behavior or managed
false. Known content attributes are filtered from primary OTLP, file, and console span exports whencaptureContentis false; local debug retention is separate. The follow-up retains these contracts rather than weakening them.Manual testing
macOS: reuse the temporary-policy script and console OTel collector
Setup
Use the single-script collector from #336701. It accepts OTLP HTTP/JSON, prints traces/logs/metrics without request headers, and creates a temporary machine-wide managed-settings file with guarded cleanup.
/Library/Application Support/GitHubCopilot/managed-settings.json. A fresh profile does not isolate machine or account policy. Do not alter an organization's policy to run this test.COPILOT_OTEL_ENABLED; setting it tofalsecan prevent policy enablement. Leavegithub.copilot.chat.otel.captureIdentityat its defaultnulland unsetCOPILOT_OTEL_CAPTURE_IDENTITY. Sign in, choose Local, and send a harmless request while OTel is still off.The gist installs the lowest-priority file source. A server telemetry block, even empty, or any observed declared native telemetry key can shadow it. Empty/unknown-only native objects are not observable by the native watcher. No exported telemetry is a failed/inconclusive test, not evidence that identity filtering worked.
Enable identity without content
Download and review
otel-restart-test.cjs. In your downloaded copy, replace only itspayloaddeclaration with:Run it in your own terminal:
Press Enter after Copilot has activated. The script applies the policy eight seconds later and keeps it for 120 seconds. Late enablement can trigger the bounded extension-host recovery from #336701. If active work vetoes recovery, finish that work and reload; do not infer successful export merely from Settings showing the policy.
Once Local is ready, send a harmless request such as
Reply with IDENTITY_TEST_OK. Also make a harmless request that actually invokes a subagent; confirm a child invocation occurred before interpreting subagent coverage. Allow approximately ten seconds for export batching.Expected collector output
Expect
POST /v1/tracesfollowed by OTLP JSON. With the gist's unchanged service configuration, identify Local extension output by resourceservice.name: "copilot-chat"andscopeSpans[].scope.name: "copilot-chat". Itsinvoke_agentspans also carrycopilot_chat.session_id.The authenticated account name appears as
user.nameon completed top-level and actual subagent invocations. OS account and host are resource attributes, not necessarily span attributes.This is an illustrative reduced payload, not a literal transcript or an assertion that each export batch contains all these fields:
{ "resourceSpans": [{ "resource": { "attributes": [ { "key": "service.name", "value": { "stringValue": "copilot-chat" } }, { "key": "process.user.name", "value": { "stringValue": "<OS account>" } }, { "key": "host.name", "value": { "stringValue": "<machine name>" } } ] }, "scopeSpans": [{ "scope": { "name": "copilot-chat" }, "spans": [{ "name": "invoke_agent <agent name>", "attributes": [ { "key": "gen_ai.operation.name", "value": { "stringValue": "invoke_agent" } }, { "key": "copilot_chat.session_id", "value": { "stringValue": "<session ID>" } }, { "key": "user.name", "value": { "stringValue": "<authenticated account name>" } } ] }] }] }] }user.namecomes from the current authenticated GitHub session'saccount.label, not a newly introduced numeric ID or email attribute. Do not count an embedded runtime's span as proof that Local's own instrumentation passed.With
captureContent: false, known external Local span content attributes such asgen_ai.input.messages,gen_ai.output.messages,gen_ai.system_instructions,gen_ai.tool.definitions,gen_ai.tool.call.arguments, andgen_ai.tool.call.resultmust be absent. Prompt/response content such asIDENTITY_TEST_OKmust not remain in those fields or the filtered content-event attributes. Events themselves may remain. This is not a claim of universal payload sanitization, and local debug retention is separate. Collector output intentionally contains identifying data in this test; review it before sharing.Negative and lifecycle cases
Finish each script run and reload/close the editor before starting a fresh test session. Change the downloaded script between runs, never the live managed-settings file; its cleanup deliberately refuses to remove a modified file.
capturefrom the test payload; no personal/environment identity opt-inuser.name,process.user.name, orhost.namein Local exports.capture: { identity: false }; set personalgithub.copilot.chat.otel.captureIdentity: trueand launch the editor withCOPILOT_OTEL_CAPTURE_IDENTITY=truecapture; launch the fresh editor withCOPILOT_OTEL_CAPTURE_IDENTITY=trueidentity: true,captureContent: falseuser.name.Cleanup: The script removes only its own unchanged test file after 120 seconds or Ctrl+C, then stops the collector. Reload or close the test editor after each run. If the script is forcibly killed or its file changes, inspect the remaining file manually; do not delete unrelated policy.
Observe withdrawal without losing the collector: let the identity-enabled script finish, confirming
Test policy removed; original absence restored.Do not reload the editor. Restart only its exported collector helper, withoutsudoor policy installation:node -e 'const { createCollector } = require("./otel-restart-test.cjs"); createCollector().listen(4318, "127.0.0.1", () => console.log("Collector ready; no test policy installed"));'Wait until the editor reflects policy removal, then send another harmless Local request without a personal/environment opt-in. The existing exporter can continue sending allowed telemetry, but the three governed identity attributes must be absent from new Local output. Ignore pre-removal batches when assessing this case. Stop the collector with Ctrl+C and reload/close the editor afterward. Do not edit the active test policy to simulate revocation.
Validation and remaining draft work
The final follow-up tree, committed as
6cb2923dcef, was validated with the repository's normal runners on Node 24.18.0:git diff --checkpassed.This supersedes the earlier Node 26 runner and follow-up lint blockers. The native matrix now tests representable declared keys instead of claiming a real native empty-object signal.
Targeted test commands
Use Node 24.18.0. From
extensions/copilot:From the repository root, with the required source/build paths present:
The account and diagnostics suites above ran with the Node runner, not in an Electron/browser session. Exporter cases use recording exporters through the real service/wrapper initialization path, not a live collector.
Still outstanding: live full-editor/OS-policy testing and the manual collector flow above, full Policy Diagnostics UI/transport verification, policy-export generation, and the Git-LFS-dependent generic tool-loop suites. Prior full core typechecking was blocked by missing Electron prerequisites; it was not rerun as a passing check for this follow-up. Targeted passes are not a claim that the full build, live rollout, or all CI checks pass.
Revocation cannot recall already persisted/exported data or cancel a batch already handed to transport. Enabling identity still uses the existing reload/recovery lifecycle.
The PR remains draft. Current agent commits are unsigned; review and a user-signed approval checkpoint remain outstanding.