Skip to content

Add per-operation timing instrumentation for historically slow/deadlocked operations #4589

Closed as not planned

Description

@all-hands-bot

Problem

Recent perf/deadlock regressions in the agent server were each caught by direct code review or a production incident rather than by an automated signal that says "this operation is taking too long / is stuck." For the operations that have historically wedged or slowed down, we have no per-operation timing that would let us (a) alert when one degrades, or (b) distinguish "slow" from "deadlocked."

We already emit coarse, privacy-bucketed conversation telemetry (see openhands-agent-server/openhands/agent_server/telemetry/), but it only captures whole-conversation duration_bucket and has no per-operation latency, so it cannot produce p50/p95 for the hot paths below.

Desired Behavior

Add per-operation timing instrumentation to the agent-server/SDK hot paths with a numeric latency property that can be percentiled (p50/p95/p99) in PostHog and, ideally, also emitted as OpenTelemetry/OpenInference spans. Timing should be recorded as wall-clock duration, and a "still in-flight / stuck" signal (or an explicit timeout) should be emitted so a deadlock is distinguishable from a merely slow operation.

Note: this is a deliberate change from the current bucketized/privacy-constrained telemetry policy (models.py / sanitizer.py), so it needs an explicit decision on how raw numeric durations are allowed through the allowlist without reintroducing a re-identification vector. Reuse the existing telemetry subscriber/sink plumbing rather than adding a parallel path.

Acceptance Criteria

Each operation below has previously been shown to deadlock or be very slow. Instrumentation must cover all of them, with a named latency metric per operation, and each historical failure must be reproducible as a failed/budget-exceeded threshold against a cheap fake-LLM or controlled-delay harness.

References

This issue was created by an AI agent (OpenHands) on behalf of the requester.

Activity

  1. github-actions commented on Aug 23, 2026

    @github-actions
    Contributor

    ✅ ready-for-dev label applied

    This issue meets the readiness criteria and is ready for contribution.
    A contributor can now open a PR that links this issue (Fixes #<number>).

    This is an automated check — no AI was used to generate this comment.

  2. jdlake commented on Aug 24, 2026

    @jdlake

    Please can the following elements of this issue be placed in another issue?

    • event_search / bash_event_search — sync executor scans; needs a sync-timer variant plus a decision on the count/fan-out magnitude (raw counts are re-identifying).
    • llm_call / switch_llm config serialization — the SDK LLM has no telemetry tap today; needs a cross-package sink hook and a contention/lock-wait timer (wire time =/= lock-wait time).
    • Lease renewal / autosave / pub/sub / condensation — background loops and cross-module fan-out; needs a defined per-iteration operation boundary and the same cross-package hook.

    My PR addresses all others.

    If you'd prefer me to update the PR with these addressed, and iron them out, let me know.

  3. all-hands-bot commented on Sep 29, 2026

    @all-hands-bot
    CollaboratorAuthor

    [Automatic Post]: This issue was automatically closed due to 36 days of inactivity. We do this to help keep the issue tracker manageable and focused on active issues. If this is still relevant, feel free to reopen it.

    Generated by OpenHands automation on behalf of the user.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestperformancePerformance issueready-for-devIssue meets development readiness criteria

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions