Skip to content

Turn on Workers Logs and write server events as allowlisted JSON lines - #188

Draft
sdcarlson wants to merge 4 commits into
mainfrom
telemetry/flyspace-logs
Draft

sdcarlson wants to merge 4 commits into
mainfrom
telemetry/flyspace-logs

Conversation

@sdcarlson

@sdcarlson sdcarlson commented Oct 31, 2026 •

Copy link
Copy Markdown
Collaborator

Hold before merging.

  • Seth's handle is required. wrangler.jsonc floors at keys (MasterMind tools/ledger/policy.json, **/wrangler.jsonc; cli.mjs floor origin/main...HEAD prints keys: wrangler.jsonc). No agent admits this PR.
  • Do not merge before the syberlabs.io privacy-copy PR (SyberLabs/SyberLabs.github.io, branch telemetry/privacy-copy), which discloses that the preview keeps Cloudflare logs for up to 7 days. This PR now keeps request logs off (below), so no client address should be kept, but the event lines are still logs and the disclosure goes live first.
  • This PR is a draft. The coordinator converted it so .github/workflows/codex-feedback.yml, which runs gh pr merge --auto --squash on a non-draft PR after answering Codex findings, cannot arm auto-merge. Seth marks it ready for review when he admits it. No agent marks it ready.

Update 2026-11-04: main merged in (b8564bf). Main's a9d4412 ("Delete the server backends no live deployment reaches") removed the LLM, inference-ledger, Postgres, hosted-auth and registry-intent backends and left their routes as 503 stubs, so 8 of the 13 converted log calls in the table below no longer exist. The conflicts kept main's deletions and stubs exactly. What this PR changes now (10 files, +459/-44):

  • wrangler.jsonc (request logs off), telemetry.ts and telemetry/events.json: unchanged.
  • Four call sites remain, all flyspace.source.failed: three in the Polymarket route, one in the public proxy's failure path. Main added no logging of its own.
  • Tests: 93 files, 855 pass (main alone: 92 files, 840). The added 15 are telemetry.test.ts. The inference-runs, auth, ledger, LLM and registry-intent tests named below went with the deleted code. Typecheck, lint, build, wrangler deploy --dry-run, e2e (7), the bundle secret scan and the audit pass locally and in CI.
  • Follow-ups, deliberately not in this merge: events.json still lists 8 events nothing writes any more (registry_intent.failed, llm.failed, the four ledger.* events, auth.unavailable, db.idle_client_failed); pruning them means rewriting the telemetry tests that use them. The telemetry.v1 contract's response test (every response byte-identical with the logger present, absent and throwing) is not here yet; the closest are the throwing-output test and the exact failure-body checks in the Polymarket and public-route tests.

The table and test list below describe the PR as first opened.

Objective. Turn on Workers Logs for the public preview with request logs off, and make every server log line a JSON event from a closed allowlist (telemetry packet P3).

What changed.

  • wrangler.jsonc: "observability": { "enabled": true, "logs": { "invocation_logs": false } }. Head sampling stays at its default, 100%. The vinext build carries both settings into dist/server/wrangler.json.
  • src/core/services/server/telemetry.ts (new, server-only). logEvent(event, fields) writes one line, {"event":"flyspace.<area>.<outcome>", ...fields}, and reads telemetry/events.json. An event not named there is not written. A field not listed for that event is never read. A value off a field's list becomes other where the list has it and is dropped where it does not. status must be an integer from 100 to 599. errorKind(err) reduces an error to its class name from a closed list (AbortError, TimeoutError, TypeError, SyntaxError, ProviderResponseError, Error, other). No message, stack, URL, body, key, prompt, excerpt, canvas or question is ever written. Errors go to console.error, the rest to console.log. It never throws.
  • telemetry/events.json (new): version 1, dataset syberlabs_events, the field lists, and nine events, each with owner Mateo, a one-line decision, a level and its fields.
  • The twelve server console calls, plus one more found on the server, now call logEvent:
was now level
[api/inference-runs] ledger query failed flyspace.ledger.query_failed, query runs error
[api/inference-runs/lineage] query failed flyspace.ledger.query_failed, query lineage error
[api/llm] provider call failed flyspace.llm.failed, llm_provider, status, kind error
[ledger] stale hosted run reconciliation failed flyspace.ledger.reconcile_failed, kind error
[ledger] could not open run (pausing 30000ms): <message> flyspace.ledger.open_failed, kind error
[ledger] could not close run <id>: <message> flyspace.ledger.close_failed, kind error
[auth] identity provider unavailable or misconfigured flyspace.auth.unavailable, reason config or timeout error
[db] idle client error (src/core/db/client.ts) flyspace.db.idle_client_failed, kind error
[registry-intent] upstream failure flyspace.registry_intent.failed, reason warn
[Polymarket API] ... (three calls) flyspace.source.failed, provider polymarket, reason, status, kind warn
[public proxy] upstream failure flyspace.source.failed, provider (the proxy id), reason, status, kind warn
  • DEPLOYMENT.md: a short "Logs" section. Request logs are off; from the app's code only the allowlisted event lines are kept, for 3 days on Workers Free and 7 on Paid; request logs go back on only after the privacy page says so.
  • Tests: telemetry.test.ts (15) and src/app/api/inference-runs/route.test.ts (2) are new, plus one auth test and log-line assertions in the existing route, ledger and proxy tests.

What did not change. Every response: status, body and headers. The failure bodies are asserted by exact equality in tests that passed against main before any route was touched (src/app/api/polymarket/route.test.ts, src/app/api/inference-runs/route.test.ts); the registry-intent test now asserts its 502 body exactly too. Which calls are errors and which are warnings: each event's level in events.json keeps it. Client code: src/core/gateway/ApiGateway.ts runs in the browser (reached only through useOmniData in client blocks), so its console call stays. No dependency, lockfile, workflow or deploy script changed. Nothing was deployed.

What the log no longer carries, on purpose. The ledger's scrubbed error text in the open and close lines, the run id in the close line, Polymarket's top-level upstream keys, and the host a public-proxy redirect pointed at. Failed and uncertain ledger rows still store the scrubbed error text.

Analytics Engine: not added. The brief writes flyspace.source.failed to an EVENTS binding only if this setup already exposes Worker bindings to route handlers. It does not:

  • vinext 1.3.0's App Router entry calls the RSC handler with (request, ctx), so env never reaches a route handler. The code reads configuration only from process.env, which carries text variables, not a binding object.
  • The one path vinext documents is import { env } from "cloudflare:workers", a workerd built-in that nothing here uses. With that one import added to a route: tsc --noEmit fails (TS2307, no Workers types are installed), next build (the Playwright harness) compiles but fails its type check the same way, and vitest cannot load the module. Making it work needs a type declaration, a vitest stub and a guard for the Node harness, which is new plumbing, not an existing binding.

So this PR writes JSON lines only and adds no analytics_engine_datasets entry.

Where this departs from the brief.

  • provider lists the sources the server actually relays: polymarket, the seven public-proxy ids (usgs, wikipedia, openlibrary, github, crossref, openmeteo, frankfurter) and other. World Bank, CoinGecko and Hacker News are fetched by the browser directly (src/core/gateway/normalizers/), so the server never sees them fail. A test checks every proxy id is listed.
  • reason also has redirect and too_large, so the proxy's six failure codes stay distinct, and config for auth. kind also has TimeoutError (what AbortSignal.timeout throws) and ProviderResponseError (the LLM adapters' own class).
  • Warn-level events print with console.log, following "errors to console.error, the rest to console.log".
  • The LLM provider is its own field, llm_provider, so it never mixes with source providers. events.json has a top-level fields map beside version, dataset and events.

Request logs are off, and why. A Workers Logs invocation log is one entry per request, with its method, full URL (query string included) and request metadata, which can include the client address. On this preview that would keep every visitor's address for days, along with the latitude and longitude a visitor sets for Open-Meteo in the /api/public query string. None of it is needed to count the events this packet exists for. With invocation_logs: false, Workers Logs keeps the allowlisted event lines from the app's code and no per-request entries. Cloudflare's Workers Logs page documents that setting. It documents no query-string redaction, so redact_query_string is not used.

Tests run (local, Node 22.22.0). npm run typecheck: 0 errors. npm run lint: 0 errors, 5 warnings, all existing and none in touched files. npm test: 142 files passed, 1 skipped; 1674 tests passed, 29 skipped (Postgres). main was 140 files and 1656 tests passed, the same 29 skipped. npm run build and CI's wrangler deploy --dry-run: pass; the allowlist is in the server bundle only, not dist/client, and the generated dist/server/wrangler.json has "observability": {"enabled": true, "logs": {"invocation_logs": false}}. npm run build:next: passes. After the request-logs change, typecheck, npm test (same counts) and the build plus dry run were run again and pass. Not run locally: test:e2e (no Playwright browser here), scan:bundle (needs CI's canary build) and the Postgres integration job. CI runs all three.

Grounding. The GitHits server did not connect in this session, so no GitHits grounding ran. The installed packages' own files were the evidence instead: the vinext 1.3.0 README ("Cloudflare Bindings") and dist/server/app-router-entry.js, and wrangler 4.152.0's config-schema.json (Observability). Cloudflare's Workers Logs page gave retention (3 days on Workers Free, 7 on Paid), the observability.enabled setting, and observability.logs.invocation_logs: false, describing an invocation log as the request, response and related metadata.

Claim ceiling. Implemented and tested, with stubs and fixtures. Not deployed: no line has been seen in Workers Logs. Cloudflare's docs show fields extracted from logged objects and do not say whether a JSON string line is parsed into fields. These lines are searchable text either way. The docs also do not say what metadata Workers Logs attaches to a console line, so whether a kept line carries any request data is unverified until a deployed line is checked. vinext and React can still write their own error lines (some include a request path), and the runtime still records an uncaught exception; neither is on the allowlist.

Workflows. ci.yml does not deploy. deploy-preview.yml deploys after CI passes on a push to main, but only while the repository variable DEPLOY_PREVIEW is true. All 37 of its runs so far were skipped, the latest on 2026-10-30 after #187, so a merge does not deploy today and logs begin at the next hand deploy. If an admin turns that variable on, the merge itself would deploy.

Architectural decisions. telemetry/events.json is the contract for server log output. The helper reads it, and a test fails on any console call in src/app/api, src/core/services/server or src/core/db outside telemetry.ts.

Falsifiable by. A response that differs from main's for the same input. A log line holding anything but an event name and allowlisted values. A server console call outside telemetry.ts.

Known limitations. A Polymarket non-JSON body logs reason network with kind SyntaxError, because reason mirrors the upstream_unreachable code the route returns. Lines carry no release or build id. Under next start they go to stdout and stderr with no collector.

Next permissible packet. Once the privacy copy is live, Seth admits this PR and marks it ready. After it merges, Seth deploys by hand (npm run deploy:vinext from a clean main) and checks Workers Logs: a flyspace. event line is there, no per-request (cf-worker-event) entries are, and the line carries no URL or address. Separately, an Analytics Engine packet once the cloudflare:workers plumbing above is decided.

Class: keys
Agent-platform: claude
Reviewed-by:
Admitted-by:
Gates-affected: none
Evidence: tested

https://claude.ai/code/session_01VHrJ4hmTtR1RFNpc215EK7


Generated by Claude Code

wrangler.jsonc enables Workers Logs. The twelve server console calls, and
the Postgres idle-client handler, now go through
src/core/services/server/telemetry.ts: one JSON line per event, fields
from telemetry/events.json only, errors reduced to their class name.
Every response is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VHrJ4hmTtR1RFNpc215EK7
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-31T02:16:41.597401Z 0727abc PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

The Logs section said "server code"; the test scans src/app/api,
src/core/services/server and src/core/db, so say that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VHrJ4hmTtR1RFNpc215EK7
… logged

observability.logs.invocation_logs is false: Cloudflare keeps no
per-request entry with a URL, query string or client address. The Logs
section of DEPLOYMENT.md says so, with retention of 3 days on Workers Free
and 7 on Paid.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VHrJ4hmTtR1RFNpc215EK7
… stubs

main moved 9 commits (to d1d82d1). Its a9d4412 deleted every server backend
the public preview cannot reach and left five routes as one-line 503 stubs,
so most of the server console calls this branch turned into logEvent calls
no longer exist. main's structure and behavior are kept exactly. Event lines
stay where failing server code still exists: the Polymarket route and the
public proxy, both flyspace.source.failed.

Conflicts (11)

Deleted on main, modified here. Resolved by keeping main's deletion:
- src/app/api/inference-runs/[id]/lineage/route.ts
  (flyspace.ledger.query_failed, query lineage)
- src/app/api/llm/route.test.ts
- src/app/api/registry-intent/route.test.ts
- src/core/db/client.ts (flyspace.db.idle_client_failed)
- src/core/services/server/auth.ts (flyspace.auth.unavailable)
- src/core/services/server/auth.test.ts
- src/core/services/server/inference.ledger.ts (flyspace.ledger.open_failed,
  close_failed and reconcile_failed)
- src/core/services/server/inference.ledger.test.ts

Both sides changed the whole file. Resolved by taking main's 503 stub
verbatim:
- src/app/api/inference-runs/route.ts
- src/app/api/llm/route.ts
- src/app/api/registry-intent/route.ts
Each stub always answers 503 and has no failure path, so there is no
equivalent point for flyspace.ledger.query_failed (runs),
flyspace.llm.failed or flyspace.registry_intent.failed.

Merged without conflict: src/core/services/server/public.providers.ts
(main's narrowed exports plus this branch's logEvent calls) and
DEPLOYMENT.md (this branch's Logs section now sits under main's
"Deploying the preview").

Other changes the merge requires
- Delete src/app/api/inference-runs/route.test.ts, added on this branch. It
  mocks the deleted ledger and db modules and imports the deleted lineage
  route. main's src/app/api/public-demo.test.ts asserts the stub's exact
  503 body.
- telemetry.test.ts: drop the import of the deleted llm.adapters module and
  its one ProviderResponseError assertion, and drop the deleted src/core/db
  from the console-call scan.
- DEPLOYMENT.md: the Logs section no longer names src/core/db.

telemetry/events.json and wrangler.jsonc are unchanged. Eight of the nine
allowlisted events now have no emitter; pruning them is left to the owner.

Checks on this merge (Node 22.22.0): typecheck 0 errors; npm test 93 files,
855 tests passed (main 92 and 840); lint 0 errors and the same 4 warnings
as main; vinext build and wrangler dry run pass with invocation_logs false
in dist/server/wrangler.json; build:next passes; test:e2e 7 passed; canary
build and scan:bundle pass; npm audit --omit=dev --audit-level=high passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VHrJ4hmTtR1RFNpc215EK7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants