[Task]: Add a /metrics endpoint to the orchestrator for observability

[grantfox-oss]

The orchestrator runs long-lived on Render and processes tasks asynchronously, but there is no way to observe its health beyond tailing logs. Basic operational questions — how many tasks are in flight, how many completed vs failed, how long steps take, how much USDC has been released — currently require scraping logs or querying the vault. That makes it hard to build a status dashboard or an alert, and hard to notice a slow degradation.

Goal

Add GET /metrics exposing process- and task-level counters, and wire the counters into the task/step lifecycle so they reflect reality.

Requirements & constraints

• At minimum expose: process uptime; task counts (total / active / completed / failed); step counts (executed / failed / timed-out); total USDC released; and a step-duration summary (at least p50 / p95 / max).
• Counters live in a small in-process module and are incremented from the executor and the task lifecycle. Seed from the existing stores where possible (orchestrator-store.ts, task-results.ts, activity-store.ts) so a restart doesn't zero everything that's already knowable.
• Keep it dependency-free for the first version — a plain counters object with increment helpers. Do not pull in a heavyweight metrics library.

Design decisions left to the implementer

• Output format. Plain JSON is the least-effort and easiest for the dashboard to consume; Prometheus text exposition is a reasonable alternative if you want Grafana compatibility. Pick one, document it, and keep the shape stable.
• How step durations are sampled/stored to compute percentiles without unbounded memory (e.g. a bounded reservoir or a fixed-size ring).

Edge cases to consider

• Concurrent tasks incrementing counters (JS is single-threaded, but async interleaving still needs correct increment ordering).
• A task that fails mid-flight must move out of active and into failed, not linger in both.

Acceptance criteria

• GET /metrics returns the counters above plus uptime, in the chosen format
• Counters increment on task submit / complete / fail and step execute / timeout
• usdc_released_total reflects released payments
• Step-duration summary (≥ p50/p95/max) is included and memory-bounded
• Endpoint documented in docs/development.md
• Vitest coverage for the counter module and the endpoint shape
• npm test passes

Relevant files

• packages/orchestrator/src/server.ts, packages/orchestrator/src/executor.ts, new packages/orchestrator/src/metrics.ts, docs/development.md

Notes for contributors

If you'd like to work on this, comment below so we can assign it to you.
Questions welcome — see CONTRIBUTING.md for setup
and workflow.

[daveades]

Hi, I'd like to work on this issue.

How I understand it

The orchestrator runs continuously but only speaks through logs, so nobody can answer basic questions without scraping them. The endpoint needs to expose counters that the executor and task lifecycle keep updated as they run, seeded on startup from the existing stores so a restart does not throw away what is already known.

What I will do

1. Add packages/orchestrator/src/metrics.ts as a plain counters object with increment helpers. No library.
2. Serve GET /metrics as JSON. The stated consumer is a status dashboard, JSON is what it wants, and Prometheus exposition without a client library means hand writing the escaping rules. I will keep the shape flat and stable so a Prometheus exporter can be layered on later without changing the internals.
3. Wire increments into the task lifecycle on submit, complete, and fail, and into the executor on step execute, fail, and timeout.
4. Track step durations in a fixed size ring of the most recent samples, sorted on read to get p50 and p95. Bounded by construction, and recency weighted, which is what a status dashboard wants since current performance matters more than a lifetime average. Max is tracked separately as a running value so it survives entries falling out of the ring.
5. Seed on startup from orchestrator-store.ts, task-results.ts, and activity-store.ts for task counts, failures, and released USDC.
6. Document the endpoint and its response shape in docs/development.md.

Things I will make sure of

1. Active never drifts. Rather than incrementing and decrementing three independent counters, I will keep total, completed, and failed as counters that only go up and derive active from them. A task then cannot sit in active and failed at once, because the invariant is structural instead of something the code has to remember.
2. Increments stay synchronous. Single threaded does not make this free. Reading a counter, awaiting something, then writing back loses every update that landed in between, so all mutation happens inside the helpers with no await in the middle.
3. USDC totals accumulate in integer base units, not floats. USDC has six decimals and repeated float addition drifts, which is not acceptable on a released payments figure. Formatting happens on output.
4. The duration summary carries its sample count. After a restart the ring is empty, since durations are not in the stores, and a p95 from four samples is noise. Exposing the count lets the dashboard know when to trust it.
5. Vitest coverage for the counter module, the lifecycle transitions including a mid flight failure, ring behaviour at and past capacity, and the endpoint shape.

One question. Should /metrics be public or protected. usdc_released_total is a business figure, and an open endpoint puts protocol revenue on the internet for anyone who guesses the path.

[grantfox-oss]

🦊 GrantFox — @daveades has been assigned to this issue as part of the Official Campaign | FWC26 campaign!

Next steps:

1. Open a Pull Request referencing this issue (e.g., Closes #87)
2. Your PR will be reviewed by the clevercon-protocol maintainers

Good luck! Track your progress on GrantFox.

[grantfox-oss]

🎉 This issue has been marked as completed on GrantFox as part of the Official Campaign | FWC26 campaign!

@daveades's PR #96 was approved and merged by @Bosun-Josh121.

🏆 @daveades: You earned 35 FoxPoints for this contribution! Your current tier: Explorer (118 total points). Track your full progress on GrantFox.

👏 Great work, @daveades! Keep contributing to clevercon-protocol.