Instrument Agent completion signals

[ross0x01]

Summary

• extend the existing hackerai-agent_run event with versioned, content-free Agent completion signals
• track per-request model step count, todo status counts, handled/SDK tool-failure counts, and distance from the latest tool failure
• distinguish a natural successful stop from limits, errors, and aborts
• record whether the request itself was an automatic continuation
• cover both the Next.js Agent path and the Trigger.dev long-running Agent path through the shared stream runner

Privacy

The new properties contain counts and booleans only. They do not include prompts, todo text, tool names, tool inputs, tool outputs, errors, code, targets, findings, or evidence.

Why

This gives us a baseline for deciding whether premature natural stops justify a narrowly scoped one-shot stop veto. It does not change Agent behavior.

Validation

• full pre-commit suite: 292 suites / 2,874 tests passed
• pnpm typecheck
• targeted ESLint and Prettier checks
• focused completion-signal, analytics, shared-runner, and Trigger contract tests

Summary by CodeRabbit

• New Features

  • Added completion analytics for agent runs, including step counts, todo progress, stop reasons, and tool-failure indicators.
  • Completion data is captured consistently for primary and retry runs.
  • Added support for identifying natural stops and recent tool failures.

• Privacy

  • Sensitive tool-content values are excluded from serialized completion data.

• Tests

  • Added coverage for completion signals, failure tracking, stop conditions, and analytics event recording.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
hackerai	Ready	Preview, Comment	Jul 22, 2026 1:40pm

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 9f6f14ec-9c2c-44e9-8a16-04e5053e9b3a

📥 Commits

Reviewing files that changed from the base of the PR and between f7d6391 and 9e7aa22.

📒 Files selected for processing (8)

• lib/analytics/__tests__/agent-completion-signals.test.ts
• lib/analytics/agent-completion-signals.ts
• lib/api/__tests__/agent-long-contracts.test.ts
• lib/api/__tests__/chat-logger.test.ts
• lib/api/agent-stream-runner.ts
• lib/api/chat-handler.ts
• lib/api/chat-logger.ts
• trigger/agent-long.ts

---

📝 Walkthrough

Walkthrough

Agent completion signals now track steps, tool failures, todo status, and natural stops across chat and trigger runs. The resulting metrics flow through stream context into completion analytics and PostHog agent-run events, with tests covering calculation, serialization, and event mapping.

Changes

Completion signal telemetry

Layer / File(s)	Summary
Signal model and calculation lib/analytics/agent-completion-signals.ts, lib/analytics/__tests__/agent-completion-signals.test.ts	Defines tracker and emitted signal types, records step and tool-failure state, derives todo and stop metrics, and tests sensitive-content omission.
Stream step tracking lib/api/agent-stream-runner.ts, lib/api/__tests__/agent-long-contracts.test.ts	Adds the tracker to stream context and records finished-step content through onStepFinish.
Completion event serialization lib/api/chat-logger.ts, lib/api/__tests__/chat-logger.test.ts	Accepts completion signals and serializes their version, stop, step, todo, and tool-failure fields into agent-run analytics.
Run-path integration lib/api/chat-handler.ts, trigger/agent-long.ts	Creates and updates trackers, threads them through agent streams, and builds completion signals for primary and retry analytics paths.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ChatHandler
  participant AgentStreamRunner
  participant AgentCompletionSignalTracker
  participant ChatLogger
  ChatHandler->>AgentCompletionSignalTracker: create tracker and record handled tool failures
  ChatHandler->>AgentStreamRunner: pass tracker in stream context
  AgentStreamRunner->>AgentCompletionSignalTracker: record completed step content
  ChatHandler->>ChatLogger: send built completion signals
  ChatLogger-->>ChatHandler: capture hackerai-agent_run event

Loading

Possibly related PRs

• hackerai-tech/hackerai#537: Modifies the shared agent completion telemetry pipeline.
• hackerai-tech/hackerai#873: Changes the same completion analytics argument and logging path.
• hackerai-tech/hackerai#904: Extends shared stream context and agent-run analytics plumbing.

Suggested labels: codex, codex-automation

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: adding Agent completion signal instrumentation.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/agent-completion-telemetry

---

_{Comment @coderabbitai help to get the list of available commands.}

[ross0x01]

CodeRabbit's generic docstring-coverage warning is non-blocking and not applicable as a change request here: the repository does not enforce docstring coverage for TypeScript helpers, the exported APIs are fully typed, and the two state transitions whose intent is not obvious already have focused comments. Adding boilerplate docstrings would not improve the correctness or privacy auditability of this telemetry change.