docs(telemetry): define the product event catalogue (CHOO-2806) - #487
Open
christian-mcdermott wants to merge 4 commits into
Open
christian-mcdermott wants to merge 4 commits into
christian-mcdermott wants to merge 4 commits into
Conversation
The server reports nothing about how it is used. This writes down what it should report, before any of it is built, so that the events can be added to the export path CHOO-2807 is building without reworking it. Three kinds of event, for two different questions. A daily snapshot carries the counts — users, rooms, agents, sessions, connectors — and answers "how much, and is it growing". Milestone events carry the seconds since install and answer "how fast does a new deployment reach value, and where does it stick". Lifecycle events carry mix and failure. No identifier for anything inside a deployment is ever sent: not a room, tenant, agent, user or message, and not a hash of one. Every metric asked for is a count or a duration, so the server counts locally, where it legitimately knows the ids, and reports only the total. That keeps the whole subject out of a privacy review and matches the rule the Console already holds itself to. Deliberately no per-message event: the relay path has no batching and no retry because a desktop app emits a handful of events an hour, and a busy server emits thousands of messages an hour. Message volume is a count in the snapshot instead, which supports the same charts. Refs CHOO-2806. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…yments Two definitions settled with the collaborating dev. The headline room figure is rooms a human created. An agent that spins up ten scratch rooms for its own orchestration has not produced ten rooms of customer value, and letting the two share a number would make adoption read as whatever the agents happened to be doing that week. Agent-created rooms keep a count of their own, and the activation milestones track user-created rooms only. Where a room event can carry the distinction it does, rather than dropping agent rooms, so "do agent-made rooms ever get used" stays answerable from the same event. Time to value is measured for deployments installed from here onward. An install date guessed from the oldest row in an existing database would be wrong by an unknown margin in an unknown direction, and a funnel built on it would read as confident when it is not. A deployment whose id is generated against a database that already has content emits no milestone events at all; the daily counts are unaffected and every deployment still reports those. Refs CHOO-2806. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…O-2806) The server reported nothing about how it is used. This adds the product half of telemetry: a daily snapshot of counts, a set of once-ever activation milestones, and the lifecycle events around rooms and connectors. Off unless an operator switches it on. Everything goes through one catalogue that declares each event and the exact properties it carries, and the catalogue is enforced rather than documented: an undeclared event, an undeclared property or a value outside its closed set all raise where the event is built. That is what makes "no room, tenant, agent, user or message is ever identified" a property of the code. The test suite plays the Console's own trick of smuggling a room name into every event and asserting it never reaches the wire. Counts are gathered per tenant and summed into deployment totals. Not because tenants are reported separately — they deliberately are not — but because row-level security means a session with nothing bound reads nothing, so a single unscoped COUNT would answer zero on a correctly configured deployment and the snapshot would be a page of confident zeroes. Turns rather than senders for the conversation figures. A message classified by who sent the one before it in the same room is the only way to tell an agent answering a person from two agents talking to each other; a sender-only count reports both as "from an agent". Room activation is derived in the daily pass rather than detected at write time. Doing it in the message path would mean a per-room flag and three extra queries on the hottest path in the server to learn something nobody needs within a day. There is deliberately no per-message event either: the relay has no batching and no retry because it was built for a desktop app emitting a handful of events an hour. Time to value is measured only for deployments installed from here onward. The migration decides that once, while the database is still empty enough to answer it honestly; an install date inferred from the oldest row would be wrong by an unknown margin in an unknown direction, and every activation figure built on it would look confident and be false. Such deployments report the daily counts and no milestones. The three bookkeeping tables carry no tenant, so they are named in GLOBAL_TABLES and in the raw-session allowlist: each records a fact about the installation rather than about anything inside it, and a deployment running three tenants has one identity, not three. Refs CHOO-2806. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Six blocking defects, all in the numbers rather than the design. Sixty-six candidates were raised across six reviewers and twenty-five survived refutation; these are the ones that changed behaviour. **The per-tenant fan-out never named its tenant.** Every query leaned on row-level security to narrow it, and the policy is exactly what does not apply on an owner connection — so a deployment with N tenants reported N times its size, and `room_became_active` fired once per tenant per room. `db/tenant_lookup.py` states this rule and every other fan-out in the tree follows it. The single-tenant tests could not fail on it, because with one tenant N is one; there is now a two-tenant regression test, and reverting one predicate makes it fail. **The activation milestone was wrong four ways at once**, and it fires once, so every wrong value was permanent. It measured from the moment the snapshot happened to run rather than from the interaction — always late, always in the same direction. It picked the fastest-activating room in the batch rather than the earliest. It did not filter to user-created rooms, so an agent's scratch room could consume the deployment's activation. And it claimed the milestone outside the enabled gate, so a deployment with telemetry off spent the claim and could never report it after opting in. Nothing exercised the method; it now has its own test file. **A telemetry lookup could fail a committed room creation.** The bridge query sat inside the argument list, so it was evaluated before the guard meant to contain it. Resolved once, ahead of the block, and the helper itself no longer raises. **`room_users_mean` could exceed `room_users_max`** — the numerator counted every room and the denominator only user-created ones. Impossible for any one population, so it read as a broken metric. **The headline room count included bridge-adopted channels.** This change introduced a third origin and the filter said "not agent", so on a Slack-heavy deployment "rooms a person created" was dominated by channels Switch was merely invited to. Split three ways. **"When telemetry is off nothing is collected" was false**, including in the Helm chart a customer reads: the snapshot loop ran regardless. It is now gated, and the interval is validated whether or not reporting is on, since zero would have meant "never not due" rather than "disabled". Also: an agent archiving a room reported nothing, because it wrote to the store directly and bypassed the service that reports; the two bridge events now share one reason set, so a classifier value only one of them declared can no longer fail validation at the moment a bridge drops; `stop()` no longer reports a disconnection for a bridge that never connected, and `restart()` says so; and the design note has been corrected to match the code in the five places it had already drifted. Refs CHOO-2806. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Writes down what the Switch core server should report about how it is used —
before any of it is built — so the events can be added to the export path
CHOO-2807is building without reworking it.Documentation only. No code, no dependency, nothing emitted.
Why now
CHOO-2806andCHOO-2807overlapped and were split: 2807 owns the sharedplumbing and the export path, this owns the product/usage events that ride on
it. The plumbing lands first, so the useful thing to do here is fix the
taxonomy — which also tells 2807 what its path has to support. The last section
of the note is that list.
The design
Three kinds of event, for two questions.
connectors — and answers "how much, and is it growing".
new deployment reach value, and where does it stick".
often a bridge drops, whether one connector fails repeatedly before it works.
No identifier for anything inside a deployment is ever sent — not a room,
tenant, agent, user or message, and not a hash of one. Every metric asked for is
a count or a duration, so the server counts locally, where it legitimately knows
the ids, and reports only the total. That matches the rule the Console already
holds itself to and keeps the whole subject out of a privacy review. The cost is
that per-room and per-tenant breakdowns are not available: we can say a
deployment had 40 active rooms this week, never which.
A per-deployment id is required — the relay admits a sender on a client id
and nothing else, and the server has none today. The row holding it also holds
the install date that every time-to-value metric is measured from.
Decisions worth reviewing
it was built for a desktop app emitting a handful of events an hour; a busy
server emits thousands of messages an hour. Message volume is a snapshot count
instead, which supports the same charts without the load or the content risk.
membership, and not two agents talking to each other.
separately so an orchestration spinning up scratch rooms cannot read as
adoption.
guessed from the oldest row of an existing database would be wrong by an
unknown margin in an unknown direction; those deployments emit no milestone
events rather than a confident-looking wrong number. Their daily counts are
unaffected.
deployment may be a customer's and the usage may be theirs.
Not in scope
The export path, the emit call, the deployment id, the consent setting and the
snapshot task all belong to
CHOO-2807or follow it. Nothing here is wired up.Refs CHOO-2806.
🤖 Generated with Claude Code