[anth-volk]

Fixes #15

Hold until the collector's deployment switch-over is verified. After that, every merge to main deploys staging and then production. This PR also needs the manual steps below done in both environments first.

Summary

• Signing. Once TELEMETRY_JWT_SIGNING_KEY holds an EC P-256 private key, the token exchange signs session tokens with ES256 and names the key in the kid header. Without it, tokens stay HS256, exactly as today.
• Public key. GET /.well-known/jwks.json publishes the public key as a JSON Web Key Set, without credentials.
• Verification during the switch. The collector accepts ES256 tokens from its key, and accepts HS256 tokens for as long as TELEMETRY_JWT_SECRET stays configured.
• Scopes. Each token carries exactly one scope, telemetry:write or artifacts:submit, and each endpoint requires its own. This uses the same code as Accept artifact submissions into private storage and run worker jobs #12.
• Deployment. Every revision mounts the key from a new Secret Manager secret, named by a new GitHub environment variable. Deployment checks that the secret exists before changing traffic.

Design

Part	Decision
Key	An ES256 (ECDSA P-256, SHA-256) private key, unencrypted PEM (PKCS #8 or SEC 1), in a new Secret Manager secret per environment. Cloud Run exposes its latest version to the service as TELEMETRY_JWT_SIGNING_KEY. Other curves, RSA, Ed25519, public keys and encrypted keys stop startup, and the error never includes the value.
kid	The RFC 7638 SHA-256 thumbprint of the public key, derived at startup rather than configured. It can't disagree with the key, and a new key always gets a new ID.
JWKS path	GET /.well-known/jwks.json returns {"keys": [{"kty": "EC", "crv": "P-256", "x", "y", "kid", "use": "sig", "alg": "ES256"}]} with Cache-Control: public, max-age=300. Before a key is configured, keys is empty. Like every route except /health, it answers 503 while the maintenance revision serves.
Issuing	ES256 whenever the key is configured; HS256 with TELEMETRY_JWT_SECRET otherwise (local development, and #12's tests).
Verification	The token header's alg chooses the key, and PyJWT then allows only that algorithm. ES256 needs the configured key's kid; HS256 needs TELEMETRY_JWT_SECRET to be set. Everything else gets 401, including none, an unknown or missing kid, a token signed by another key under our kid, and an HS256 token keyed with the published public key. 401 matters: the emitter exchanges again after a 401, but stops delivering a run after a 403. Issuer, audience, expiry and the required claims are checked as before.
Switch window	Starts with the first deployment that mounts the key. Ends with a follow-up change that stops mounting TELEMETRY_JWT_SECRET. It must last at least one session-token lifetime (TELEMETRY_SESSION_TOKEN_TTL_SECONDS, 1 hour) after the last HS256-issuing revision stopped serving, because that revision's tokens stay valid that long. How much longer to keep it, as rollback cover, is the team's call.
Retiring HS256	No code change is needed. CollectorSettings accepts either the key or the HS256 secret, and requires at least one. The follow-up removes TELEMETRY_JWT_SECRET from both --set-secrets lists and removes JWT_SECRET_NAME from scripts/deploy.sh and push.yaml. After that deploys, the old variables and secrets can be deleted. The README documents these steps.
Key rotation (not needed now)	Put the new key in a new secret, give it the same two grants, point TELEMETRY_JWT_SIGNING_KEY_SECRET_NAME at it, and deploy. Don't add a version to the mounted secret: Cloud Run reads it when each instance starts, so instances of the serving revision would mix keys until the next deployment. Tokens signed by the previous key get 401 and are exchanged again.
Scopes	Identical to #12: SessionScope = Literal["telemetry:write", "artifacts:submit"] in models.py, SCOPE_REFUSALS, a scope parameter on _issue_session_token, and a required_scope parameter on _decode_session_token. The exchange still issues only telemetry:write. #12 adds the {"scope": "artifacts:submit"} exchange request and the endpoints that require that scope. Until then, nothing on main issues artifacts:submit.

Who verifies the tokens. Only the collector does.

• The emitter treats them as opaque bearer tokens.
  • CollectorSession.credential() in src/microcosm_emitter/core/auth.py keeps access_token as a string for expires_in seconds and never decodes it. Version 0.2.1, which builds pin, does the same.
  • After a 401 from registration or events, telemetry/service/collector.py calls session.invalidate() and exchanges again.
  • Emitter Build from a maintained base image without the eMudhra roots #27 keeps the token opaque and only adds {"scope": ...} to the exchange body.
• The dashboard still reads with X-Telemetry-Read-Token. Verifying consumers' Vercel OIDC tokens is a separate change (C6).

Rollout order

1. Prerequisite. Finish and verify the collector deployment switch-over: the #233 runbook, COLLECTOR_DEPLOY_ENABLED, and Carry over calibration-diagnostics#219: apply each event batch incrementally #11 verified on its release commit. COLLECTOR_DEPLOY_ENABLED isn't set today.
2. Manual steps, in staging and production (listed below).
3. Merge this PR with a merge commit, after bringing it up to date with main. The release deploys staging first, then production only if staging succeeds. Each new revision accepts HS256 and ES256 tokens and issues ES256 tokens.
4. Verify in each environment:
   • /.well-known/jwks.json lists one key.
   • Builds keep registering runs and sending events.
5. Keep the HS256 secret mounted for at least one token lifetime, and until a rollback to a revision from before this change is no longer wanted.
6. Retire HS256 with the follow-up change described above, then delete the old variables and secrets.

How this meets the plan's order ("deploy dual acceptance before anything issues ES256-only tokens"):

• One issuer. The collector is the only service that issues these tokens.
• No gap between the two. Every revision that issues ES256 tokens also accepts HS256 tokens. So in each environment, dual acceptance is live no later than the first ES256 token.
• Older revisions never receive ES256 tokens. During a deployment, the maintenance revision sits between the old revision and the new one, and it answers 503 to every data request.
• ES256-only comes last. Acceptance of ES256 alone happens only in step 6.
• Rollback is safe. A revision from before this change answers 401 to ES256 tokens. After a rollback, the emitter exchanges again, gets an HS256 token and carries on after one retry.

Environment variables and secrets

Name	Read by	Provider	Wiring	Status
TELEMETRY_JWT_SIGNING_KEY (new)	the service, at startup (CollectorSettings.from_environment)	Google Secret Manager: a new secret per environment	--set-secrets TELEMETRY_JWT_SIGNING_KEY=${JWT_SIGNING_KEY_SECRET_NAME}:latest in deploy_maintenance and deploy_candidate	Unverified: the secret doesn't exist yet (manual step).
JWT_SIGNING_KEY_SECRET_NAME (new)	scripts/deploy.sh: require_configuration, check_secrets_exist, deploy_maintenance, deploy_candidate	GitHub Actions environment variable TELEMETRY_JWT_SIGNING_KEY_SECRET_NAME in staging and production	Job-level env: of deploy-staging and deploy-production in push.yaml, so every step that runs the script inherits it	Not set: the API lists 15 variables in each environment, not this one (manual step).
TELEMETRY_JWT_SECRET (unchanged; retired later)	the service	Google Secret Manager; name from TELEMETRY_JWT_SECRET_NAME	unchanged	The variable exists in both environments. Unverified: the secret's existence and the runtime access (see below).
PROJECT_ID, RUNTIME_SERVICE_ACCOUNT (unchanged)	check-secrets-exist.sh, gcloud run deploy	GitHub environment variables TELEMETRY_GCP_PROJECT_ID, TELEMETRY_RUNTIME_SERVICE_ACCOUNT	unchanged	Present in both environments.
Runtime access to the new secret	Cloud Run, when it creates a revision	Google Cloud IAM	roles/secretmanager.secretAccessor for the runtime service account on the new secret	Unverified (manual step).
Metadata read of the new secret	gcloud secrets describe in check-secrets-exist.sh	Google Cloud IAM	The deployment service account (TELEMETRY_GCP_DEPLOY_SERVICE_ACCOUNT) needs roles/secretmanager.viewer on it, as it has on each of the other secrets	Needs a grant (manual step 3). The deployment account holds that role only through bindings on each existing secret, not project-wide, so a new secret starts without it.

What fails, and where, if a step is missing:

Missing	Where the deployment stops	Effect on traffic
The GitHub variable	"Require deployment configuration"	No Google Cloud call is made.
The secret	"Check required Secret Manager entries exist"	Traffic is unchanged.
Runtime access, or a key that isn't P-256	The maintenance revision, created with --no-traffic, never starts	Traffic is unchanged.

In each case staging fails first, and production doesn't run.

Manual steps (not done here)

For each environment, staging first:

1. Create the secret. Generate a P-256 private key straight into a new Secret Manager secret in the collector's project, without writing it to disk. For example: openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 | gcloud secrets create <name> --project <project> --data-file=-. Use a different key for each environment.
2. Grant runtime access. Give the runtime service account roles/secretmanager.secretAccessor on that secret.
3. Grant metadata access. Give the deployment service account roles/secretmanager.viewer on that secret. It has no project-wide Secret Manager role, so without this grant the deployment stops at "Check required Secret Manager entries exist".
4. Set the GitHub variable. Set TELEMETRY_JWT_SIGNING_KEY_SECRET_NAME to the secret's name in that GitHub environment.

After the switch window (follow-up): delete TELEMETRY_JWT_SECRET_NAME from both environments and delete the HS256 secrets.

Expected conflict with #12

Both PRs edit the token functions in app.py. To keep the conflict small, this PR writes the scope change exactly as #12 does, so those hunks merge on their own. I merged this branch into claude/submission-intake locally (not pushed) to find the actual conflicts:

File	Conflict	Resolution
telemetry_collector/app.py, imports	#12 adds TokenExchangeRequest to the models import; this PR adds the session_tokens import right after it.	Keep both.
telemetry_collector/models.py	Both add the same SessionScope alias; #12 also adds TokenExchangeRequest right after it.	Keep #12's class.
uv.lock	This PR adds cffi, cryptography and pycparser; #12 adds the Google client packages, which also bring in cryptography.	Run uv lock again.

Everything else merged without conflicts: _issue_session_token, _decode_session_token, CollectorSettings, from_environment, pyproject.toml and the README.

With those three resolutions, the combined code passes:

• Ruff check and format check.
• Accept artifact submissions into private storage and run worker jobs #12's test_session_scopes.py and test_intake_settings.py, together with this PR's three token test files: 95 passed.

#12's token tests construct settings without a signing key, so they still issue and decode HS256 unchanged. Once both merge, #12's artifacts:submit tokens are ES256 too.

tests/test_session_token_scopes.py (the verification side) and #12's tests/test_session_scopes.py (the exchange side) overlap slightly. Whichever PR merges second can fold them together.

Commits (red, then green)

I ran each commit's test files before the change (red) and after it (green), then again after committing. Failing tests ran against stubs that raise NotImplementedError, so each test failed on its own rather than at import.

1. Load an ES256 session signing key and derive its key ID.
   • Red: 12 failed.
   • Green: 12 passed.
2. Accept ES256 and HS256 session tokens during the signing switch.
   • Red: 37 failed, 13 passed. The 13 are commit 1's 12 tests plus test_settings_still_refuse_a_short_hs256_secret, which already held.
   • Green: 50 passed.
3. Issue ES256 session tokens and publish the JWKS once the key is configured.
   • Red: 12 failed, 51 passed. The one new test that passed, test_exchange_issues_hs256_tokens_until_a_signing_key_is_configured, checks today's behaviour.
   • Green: 63 passed.
4. Split session token scopes into telemetry:write and artifacts:submit.
   • Red: 8 failed, 2 passed. Both cases of test_exchange_still_issues_telemetry_write_tokens already held.
   • Green: 10 passed.
5. Mount the session signing key in every deployed revision.
   • Red: 7 failed, 22 passed. The 22 are the existing tests in test_deploy_script.py and test_deployment_workflow.py.
   • Green: 29 passed.

Testing

• Changed test files. Every test file this branch adds or changes, run together, passes: 102 passed. Those files are test_session_token_keys.py, test_session_token_signing.py, test_session_token_scopes.py, test_deploy_script.py and test_deployment_workflow.py. All keys are generated during the test run, and no test reads a stored key.
• Repo scripts. tools/check-quality.sh (Ruff check and format check, bash -n on scripts) passes, and tools/check-changelog.sh passes (0.3.0 to 0.4.0).
• Not run locally.
  • The rest of the suite, including test_app.py, even though app.py changed. Its settings have no signing key, so its HS256 decode still applies. CI covers it.
  • actionlint, which CI runs.

Fixes after review

A review of d992b4a found one documentation bug and two smaller gaps. Fixes, in order:

1. 808558d Rotate the signing key through a new secret, and name both secret grants. Documentation only, so there is no red step.
   • The bug: the README said to rotate the key by adding a version to its secret and then deploying.
   • Why that fails: Cloud Run reads a secret mounted as an environment variable when each instance starts. So every instance the serving revision started after the new version was added got the new key, while its running instances kept the old one.
   • Effect: until the next deployment, which needs a merge to main, a token from one instance got 401 from another, and /.well-known/jwks.json depended on which instance answered.
   • The fix: the README now rotates through a new secret and a change to TELEMETRY_JWT_SIGNING_KEY_SECRET_NAME, which leaves the serving revision on the old secret.
   • It also names the two roles the new secret needs on itself: roles/secretmanager.secretAccessor for the runtime service account, and roles/secretmanager.viewer for the deployment identity.
2. b8934c2 Test the registered-claim checks for HS256 tokens and not-yet-valid times. Test only; the behaviour already held, so there is no red step.
   • The issuer, audience, expiry and missing-claim checks now run for HS256 tokens as well as ES256.
   • New cases: nbf in the future and iat in the future.
   • A missing claim must raise MissingRequiredClaimError naming that claim.
   • Green: 22 passed for these two tests, up from 9.

Every test file this branch adds or changes, run together after the fixes: 115 passed. tools/check-quality.sh and tools/check-changelog.sh pass.

Review round 2026-10-22

A second review of b8934c2 found no code bug, so no commits were added. The earlier fixes (808558d, b8934c2) are confirmed.

Corrections to the text above:

• The deployment switch-over is done. COLLECTOR_DEPLOY_ENABLED has been true since 2026-10-20, and both services run the 9b32902 release from this repository. So rollout step 1 is complete, and merging now releases and deploys.
• Manual steps 1–4 are still needed in both environments. No signing-key secret exists yet, and neither environment has TELEMETRY_JWT_SIGNING_KEY_SECRET_NAME. If this PR merges first, the staging deployment stops at "Require deployment configuration" before any Google Cloud call, and so does every later collector deployment until the variable is set. Traffic is not changed.

Checks:

• Tests. The changed test files pass: 115 passed. On a local merge of main into this branch (not pushed), the same files plus test_app.py pass: 147 passed. tools/check-quality.sh passes.
• Conflicts.
  • main merges cleanly.
  • Give the collector's worker jobs their own Hugging Face read token #18 merges cleanly.
  • Check the collector's Hugging Face token and each dataset's releases.json daily #19 conflicts only in tests/test_deploy_script.py. Whichever PR merges second resolves it.
• Token clients.
  • Emitter 0.2.1 (the version Microcosm pins) and the emitter's main still treat the token as opaque. They exchange again after a 401.
  • microcosm-runs only forwards the token to the collector.
  • The dashboard uses X-Telemetry-Read-Token.
• Scopes with Accept artifact submissions into private storage and run worker jobs #12. All four submission routes in Accept artifact submissions into private storage and run worker jobs #12 require artifacts:submit through the same _decode_session_token signature this PR adds.

🤖 Generated with Claude Code