Skip to content

feat(console): give connector events a failure reason and a duration - #488

Open
christian-mcdermott wants to merge 1 commit into
mainfrom
feat/connector-telemetry-failure-and-duration
Open

christian-mcdermott wants to merge 1 commit into
mainfrom
feat/connector-telemetry-failure-and-duration

Conversation

@christian-mcdermott

Copy link
Copy Markdown
Collaborator

What

Two gaps in the Console's product events, both in the same area:

  1. connector_installed had an outcome and no reason. A failed connector install was a bare count with nothing to act on — the only action event of its kind in the catalogue without a failure_reason (compare agent_cli_action, agent_created, bridge_connected).
  2. Nothing in the catalogue carried a duration. Every event is stamped with wall-clock time, so deltas were computable at the far end, but sends are fire-and-forget with no retry and there is deliberately no correlation id — so pairing broke under concurrency and lost an end whenever a send dropped.

TelemetryConnectorFailure

Named at the point of failure rather than mapped from an error type: these paths only produce a message, and a message cannot be sent.

code means
none success
unsupported connector declared, nothing resolvable to manage it
marketplace_failed the marketplace would not register or re-point at its source
install_command_failed the host CLI refused the plugin
update_command_failed the host's update verb failed
uninstall_command_failed the host's remove verb failed
files_write_failed the app writes this connector itself, and the write failed
files_unimplemented the plugin declares kind: 'files' and implements no behavior
error reserved catch-all

The reinstall split is the one worth reviewing. Codex has no update verb, so an update is remove-then-add and was_reinstall is true for both halves. It cannot tell "nothing changed" from "the agent now has no connector at all" — uninstall_command_failed vs install_command_failed can, and that is the difference between a no-op and a broken agent.

The codes also separate the two kinds of connector, which was the original question behind this change. A marketplace-driven host (Claude Code, Codex) and one whose connector the app writes (OpenCode) fail in entirely different places. agent_type implies which today, 1:1 — but that mapping is a dashboard convention that breaks silently the day a second files host lands, and the failure codes do not depend on it.

duration_ms

On the three connector events and on agent_cli_action, where a CLI install taking four minutes and one taking ten seconds were otherwise the same row.

  • performance.now(), not Date.now() — monotonic, so an NTP step or a machine waking mid-install cannot produce a negative duration or an hour that never passed. A negative number in a payload is worse than no number: nothing at the far end can tell it from real data.
  • Whole milliseconds, nothing capped. A capped duration is a number the operation did not take, which is the invented value the rest of the catalogue refuses. Some of these legitimately include a password prompt somebody left on screen — read them as percentiles, not as a mean. This is documented on the type.
  • Timed around the operation alone, excluding manager resolution and SSH connect, so the first measurement of a session is not systematically different from the rest.

Nothing free-text, no new identifier: an elapsed time at this resolution names no path, host or command. The consent copy gains a line anyway — console/AGENTS.md treats widening what is sent as a consent decision rather than a code change.

Also in here

  • connector_installed uses agentTypeOf() like its siblings instead of an inline isValidProviderId ternary.
  • runFiles folds resolving into its own try on both drivers. That deletes the duplicated guard in runInstall and fixes a real bug: update and uninstall on a files connector with no behavior threw out to the UI as a stack trace with nothing reported. Only install was guarded.
  • Remote's resolveFiles carries the connector version like local's, removing three connectorVersion() call sites.
  • sentAttributes() in the telemetry service test read only stringValue, so every number and boolean in a payload read back as undefined — an assertion that a numeric property was carried would have passed against a payload that dropped it. Now unwraps all three OTLP shapes.

Testing

New why a connector operation failed block pinning each code, including both halves of the reinstall split; duration.test.ts; and the existing event assertions updated.

pnpm run format, lint, typecheck and vitest --project node --project main-db are clean. Three test failures and a handful of typecheck errors remain — all reproduce on untouched main (session-spawner, sidecar-runtime, room-connection; js-yaml types, reapOrphanedRuntimes, onRoomRejected).

Not in scope

catchUpConnectors calls switchSetupService.update(), so the automatic once-per-install catch-up emits connector_updated indistinguishably from someone pressing Update. That predates this change, but duration_ms makes it worse — background catch-ups will skew the distribution and the failure counts. The catalogue already has the pattern for it (TelemetryAgentRemoveTrigger, which exists for exactly this reason); a trigger: 'user' | 'catch_up' would sort it. Happy to add it here or separately.

Unrelated to #487, which defines a catalogue for the core server's own (not-yet-built) telemetry path. This is Console-side only.

🤖 Generated with Claude Code

`connector_installed` carried an outcome and no reason, so a failed
connector install was a bare count with nothing to act on — the one
event of its kind in the catalogue without a `failure_reason`, and the
underlying `SwitchSetupResult` carries only a message, which cannot be
sent.

Add `TelemetryConnectorFailure`, named at the point of failure rather
than mapped from an error type. The codes separate walls that need
different fixes: a marketplace that will not register, a host CLI that
refuses the plugin, and the app's own file writing. The reinstall split
is the one that matters most — Codex has no update verb, so an update is
remove-then-add, and `was_reinstall` is true for both halves. It cannot
tell "nothing changed" from "the agent now has no connector at all";
`uninstall_command_failed` vs `install_command_failed` can.

Add `duration_ms` to the three connector events and to
`agent_cli_action`, where a CLI install taking four minutes and one
taking ten seconds were otherwise the same row. Measured on a monotonic
clock so a clock step cannot yield a negative duration, in whole
milliseconds, and deliberately uncapped: a capped duration is a number
the operation did not take. Timed around the operation alone, excluding
manager and SSH resolution, so the first measurement of a session is not
systematically different from the rest.

Along the way:

- `connector_installed` uses `agentTypeOf()` like its siblings rather
  than an inline `isValidProviderId` ternary.
- `runFiles` folds resolving into its own try on both drivers, which
  deletes the duplicated guard in `runInstall` and fixes `update` and
  `uninstall` on a files connector with no behavior — those threw out to
  the UI as a stack with nothing reported.
- Remote's `resolveFiles` carries the connector version like local's,
  removing three `connectorVersion()` call sites.
- The consent copy gains a line. Widening what is sent is a consent
  decision, not a code change.
- `sentAttributes()` in the telemetry service test read only
  `stringValue`, so every number and boolean in a payload read back as
  `undefined` — an assertion that a numeric property was carried would
  have passed against a payload that dropped it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant