feat(orchestrator): add GET /metrics endpoint with task and step counters

[daveades]

Summary

Adds GET /metrics to the orchestrator so its health can be observed without
tailing logs, and wires the counters into the task and step lifecycle.

Related issue

Closes #87

Changes

• New packages/orchestrator/src/metrics.ts: a plain counters object with
  increment helpers. No new dependencies.
• GET /metrics returns uptime, memory, task counts (total / active /
  completed / failed / interrupted), step counts (executed / failed /
  timed-out), usdc_released_total, and a step-duration summary
  (p50 / p95 / max).
• Counters increment from runTask() (task lifecycle) and PlanExecutor
  (steps and vault releases).
• On startup, counters are seeded from activity-log.json and
  task-results.json so a redeploy does not zero totals that are still
  knowable.

Decisions a reviewer should weigh in on

Output format is plain JSON, not Prometheus text exposition. The dashboard
is the main consumer and already speaks JSON everywhere else. Prometheus would
have meant a bespoke serializer for a format nothing in the repo currently
reads. The shape is documented as stable.

Task transitions are keyed on task_id in a Set, not a bare counter.
This makes the "must not linger in both active and failed" requirement hold
structurally: the terminal helpers only count if Set.delete returns true, so
repeated or racing calls are no-ops. That is what makes it safe to call
taskFailed at the top of runTask's catch (before anything that could itself
throw) plus a backstop in the POST /api/tasks catch.

Step failures are counted inside makeFailedResult. All four executor
failure paths already funnel through it exactly once, so one call site covers
them all and future failure paths are counted for free.

steps.executed counts all attempts, with failed and timed_out as
nested subsets. The issue wording admits the other reading, so this is spelled
out in the docs.

tasks.interrupted is a field beyond the issue's minimum. Tasks the
activity log still shows in flight belong to a process that has exited, so
counting them as active would misreport a fresh process. Consequence,
documented: total does not equal the sum of the other four.

Durations use a 1024-entry ring buffer — bounded memory, and percentiles
track recent behaviour rather than being diluted by all history.
steps.executed keeps counting past the cap; only the sample window is bounded.

Testing

Full CI gate run locally, all green:

npm run typecheck   # all 8 workspaces
npm run lint        # clean
npm run format:check # clean
npm run build       # clean
npm test            # 115 passed | 6 skipped (121)

The 6 skips are the pre-existing vault integration tests that auto-skip without
testnet env vars.

28 of those tests are new:

• metrics.test.ts (25) — counter transitions, idempotent terminal calls,
  interleaved async completion, timeout classification, percentile math,
  ring-buffer bounding, and startup seeding.
• server.metrics.test.ts (3) — real HTTP round-trip against the Express app
  asserting status, content-type, and full response shape.

Not run against live testnet: the endpoint is pure in-process state with no
Stellar dependency.

Checklist

• npm run lint passes
• npm run typecheck passes
• npm test passes
• npm run format:check passes
• If a contract changed: N/A — no contract changes
• Docs updated if behavior, setup, or APIs changed
• If a contract's public interface or storage layout changed: N/A

Summary by CodeRabbit

• New Features

  • Added a GET /metrics endpoint providing JSON operational metrics, including uptime, task and step counts, timeout totals, released USDC, duration summaries, and memory usage.
  • Metrics now track task lifecycle outcomes, step successes and failures, execution durations, and released USDC amounts.
  • Persisted activity and task history are incorporated into metrics during startup.

• Documentation

  • Documented the metrics endpoint, response fields, behavior, and testing guidance.

[coderabbitai]

📝 Walkthrough

Walkthrough

Changes

Orchestrator metrics

Layer / File(s)	Summary
Metrics state and aggregation packages/orchestrator/src/metrics.ts, packages/orchestrator/src/metrics.test.ts	Adds task and step counters, timeout classification, USDC totals, bounded duration percentiles, uptime and memory snapshots, startup seeding, reset support, and comprehensive Vitest coverage.
Task and execution lifecycle wiring packages/orchestrator/src/executor.ts, packages/orchestrator/src/server.ts, packages/orchestrator/src/task-results.ts	Records task transitions, step outcomes, latencies, and released USDC, and seeds metrics from persisted activity and task results during startup.
Metrics endpoint and documentation packages/orchestrator/src/server.ts, packages/orchestrator/src/server.metrics.test.ts, docs/development.md	Adds unauthenticated GET /metrics JSON output, endpoint tests, and response-shape documentation.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant TasksRoute
  participant runTask
  participant executeStep
  participant MetricsModule
  Client->>TasksRoute: submit task
  TasksRoute->>runTask: start task
  runTask->>MetricsModule: record taskStarted
  runTask->>executeStep: execute step
  executeStep->>MetricsModule: record step result and latency
  executeStep->>MetricsModule: record released USDC
  runTask->>MetricsModule: record taskCompleted or taskFailed
  Client->>MetricsModule: GET /metrics
  MetricsModule-->>Client: JSON metrics snapshot

Loading

Suggested reviewers: bosun-josh121

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: adding a GET /metrics endpoint with task and step counters.
Linked Issues check	✅ Passed	The changes satisfy the /metrics observability requirements, including counters, seeding, docs, and tests.
Out of Scope Changes check	✅ Passed	The diff stays focused on metrics, endpoint wiring, docs, and tests, with no clear unrelated changes.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/orchestrator/src/executor.ts`:
- Around line 326-328: Move the stepExecuted(latency_ms) call in the successful
execution path to immediately before the successful return, after rateResponse
and step_complete complete. Keep the existing latency calculation and ensure
exceptions continue to reach stepFailed without recording a successful execution
first.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: e5abb9af-6abd-4e43-85a2-ab1747dcefed

📥 Commits

Reviewing files that changed from the base of the PR and between a02baea and 0d09b55.

📒 Files selected for processing (7)

• docs/development.md
• packages/orchestrator/src/executor.ts
• packages/orchestrator/src/metrics.test.ts
• packages/orchestrator/src/metrics.ts
• packages/orchestrator/src/server.metrics.test.ts
• packages/orchestrator/src/server.ts
• packages/orchestrator/src/task-results.ts