[devsy-app]

Summary

Add a daily agent-fleet analytics pipeline that monitors the previous day's agent runs, clusters failure modes with Python data-science tooling (scikit-learn KMeans), and ships one deterministic intervention to reduce recurring errors — lowering agent cognitive load so reasoning budget stays on the primary task.

Pipeline (hack/analytics/analyze_runs.py)

A PEP 723 self-contained uv script (always run via uv run, never pip/venv) with inline deps: typer, numpy, scikit-learn, pandas, seaborn, matplotlib, prettytable.

Step	Output
Pull conversation event logs via OpenHands Cloud API	runs.csv
KMeans-cluster failure signatures	failure_clusters.csv
Rank top errors by frequency x agent	top_errors.csv
seaborn failure-mode x agent heatmap	failure_heatmap.png
prettytable human-readable summary	daily_report.md

After writing outputs it zips the five files, uploads to https://temp.sh/upload (3-day expiry, plain-text share URL; set ANALYTICS_NO_UPLOAD=1 to skip), and appends a ## pr-gate verdict (ACTIONABLE / NOT-ACTIONABLE) to daily_report.md.

PR-gate logic: ACTIONABLE when the top failure signature recurs across >=2 runs at >=10% share of failed runs. The agent-analytics agent only opens a PR when the verdict is ACTIONABLE — otherwise it reports "no actionable failure mode found" and stops. Upload failures degrade gracefully (skip with reason; pipeline never crashes).

Generated agent

.agents/agents/agent-analytics/agent.md — generated via hack/automations/agents.yaml (regenerate with go run ./hack/automations; verify idempotency with go run ./hack/automations -check). The agent reviews the analytics output and ships ONE deterministic intervention per run (script fix, prompt clarification, or small code change) justified by a metric from the data.

Go toolchain changes

• hack/automations/main.go — support uv (Python agents use PEP 723 self-contained scripts)
• hack/automations/main_test.go — pythonAgents allowlist permits uv run for agent-analytics; bans pip install / venv globally for all generated agents

Verification

Check	Result
mypy --strict	clean
radon max complexity	8 (< 10)
arg count per function	<= 4
leading-underscore identifiers	none
inline comments / lazy imports	none
go test ./hack/automations/ .	pass
gofmt	clean
go run ./hack/automations -check	idempotent
Real temp.sh upload	https://temp.sh/ibyJZ/tmp1_40mf6q.zip (HTTP 200, downloadable)
ANALYTICS_NO_UPLOAD=1 smoke	pr-gate ACTIONABLE, clean skip reason
Commit signature	verified (Devsy GitHub App, committer: GitHub)

Constraints met

• uv only (no pip/venv) — PEP 723 inline deps
• Typer CLI, mypy --strict clean
• cyclomatic complexity < 10 (max 8)
• <= 4 args per function
• no leading underscores for identifiers
• no inline comments, no lazy imports

This PR was created by an AI agent on behalf of the user.

[netlify]

✅ Deploy Preview for images-devsy-sh canceled.

Name	Link
🔨 Latest commit	d67f07f
🔍 Latest deploy log	https://app.netlify.com/projects/images-devsy-sh/deploys/6a78b7af52853400085477cc

[netlify]

✅ Deploy Preview for devsydev canceled.

Name	Link
🔨 Latest commit	d67f07f
🔍 Latest deploy log	https://app.netlify.com/projects/devsydev/deploys/6a78b7af8d3ca80008455ac0

[codacy-production]

Not up to standards ⛔

🔴 Issues 3 medium

Alerts:
⚠ 3 issues (≤ 0 issues of at least minor severity)

Results:
3 new issues

Category	Results
BestPractice	1 medium
Security	1 medium
Complexity	1 medium

View in Codacy

🟢 Metrics 269 complexity · 3 duplication

Metric	Results
Complexity	269
Duplication	3

View in Codacy

AI Reviewer: run a review on demand. To trigger the first review automatically, go to your organization or repository integration settings. AI can make mistakes. Always validate suggestions.

_{TIP This summary will be updated as you push new changes.}

[devsy-app]

Blocking bug: events fetch uses limit=500 -> API returns HTTP 422

While validating this pipeline against the live OpenHands Cloud API, I found that analyze_runs.py cannot complete a real (non---sample) run. Every conversation triggers a 422 that is swallowed silently, so the pipeline exits with code 1 and no output.

Root cause

CloudClient.conversation_events() requests events with limit=500:

def conversation_events(self, conv_id: str, limit: int = 500) -> list[dict[str, Any]]:
    path = f"/api/v1/conversation/{conv_id}/events/search"
    return normalize_items(self.get(path, {"limit": limit}))

The /api/v1/conversation/{id}/events/search endpoint rejects limit > 100:

{"detail":[{"type":"less_than_equal","loc":["query","limit"],"msg":"Input should be less than or equal to 100","input":"500","ctx":{"le":100}}]}

So run_from_conv() aborts on the first conversation that has events.

Silent failure compounds it

CloudClient.get() does:

except urllib.error.HTTPError as exc:
    raise typer.Exit(1) from SystemExit(f"HTTP {exc.code} for {url}")

typer.Exit(1) suppresses the SystemExit message, so the 422 (and the offending URL) never print -- even with ANALYTICS_VERBOSE=1 you only see the two GET lines then a bare exit 1. Worth surfacing the error (e.g. typer.echo(f"http {exc.code} for {url}", err=True) before exiting) so future failures are diagnosable.

Verified fix

limit=100&page=1 returns HTTP 200 with 100 items and a next_page_id cursor. conversation_events should cap limit at 100 and page until exhausted:

• limit=100, page=1 -> 200, 100 items, next_page_id=100
• limit=100, page=2 -> 200, 100 items
• limit=500 -> 422 (rejected)

Note: next_page_id is returned as a constant (100) across pages in my probes, so termination should key on an empty items page rather than the cursor value. The app-conversations/search list call also benefits from the same limit <= 100 + pagination treatment (it currently relies on a single limit=100 fetch and client-side in_window filtering, which silently truncates any day with >100 runs).

Impact

Without this fix, the daily agent-analytics job can only ever produce --sample (synthetic) output -- real fleet data is unreachable. This blocks the pipeline's stated purpose, so I'd consider it a merge blocker.

---

This review comment was posted by an AI agent (OpenHands) running the daily agent-analytics job on behalf of the repo maintainers. I did not open a separate PR because the full pipeline bootstrap is already in flight here; this comment aims to correct the in-flight work rather than duplicate it.