Skip to content

Add Microsoft's AI Coach analytics: category scorecards, anti-pattern detection, backfill - #11

Merged
akhilsinghcodes merged 2 commits into
mainfrom
feature/enhancements
Jun 20, 2026
Merged

akhilsinghcodes merged 2 commits into
mainfrom
feature/enhancements

Conversation

@akhilsinghcodes

Copy link
Copy Markdown
Owner

Summary

  • New Analytics tab: per-session practice scorecards across 4 categories (Prompt Quality, Session Hygiene, Code Review, Tool Mastery), backed by all 45 detection rules ported from microsoft/AI-Engineering-Coach
  • Parses the agent's own ~/.claude/projects / ~/.codex logs, scoped to the exact AgentFleet session time window (not the repo's full history), and runs them through the rule engine to surface anti-patterns (severity, occurrences, description, suggestion) plus per-category and overall practice scores
  • New packages/ai-coach-analytics workspace package (parsers, rule engine, 45 rule + 10 metric definitions) and session_analytics table (additive only — no changes to existing tables/routes)
  • Backfill script (apps/server/scripts/backfill-analytics.ts) to analyze historical stopped sessions that predate this feature
  • Fixed a session-delete regression (FOREIGN KEY constraint failed) introduced by the new session_analytics table
  • Docs: AI_COACH.md explains the scoring formula and all 45 rules by category; NOTICE updated with attribution

Test plan

  • pnpm -r typecheck passes
  • pnpm -r lint passes
  • pnpm --filter @agents_fleet/server test passes (15/15)
  • Backfilled 13 historical sessions; confirmed distinct, non-uniform scores/anti-patterns per session via direct SQLite query (e.g. one session scored 92 with a real "Auto Model Avoidance" anti-pattern, 36 occurrences)
  • Verified session delete no longer throws FK errors
  • Manually click through Analytics tab in the browser for a few real sessions

@akhilsinghcodes
akhilsinghcodes merged commit eb9631b into main Jun 20, 2026
1 check passed
@akhilsinghcodes
akhilsinghcodes deleted the feature/enhancements branch June 20, 2026 08:32
@akhilsinghcodes
akhilsinghcodes restored the feature/enhancements branch June 20, 2026 16:27
akhilsinghcodes added a commit that referenced this pull request Jul 29, 2026
Add Microsoft's AI Coach analytics: category scorecards, anti-pattern detection, backfill
akhilsinghcodes added a commit that referenced this pull request Aug 1, 2026
Add Microsoft's AI Coach analytics: category scorecards, anti-pattern detection, backfill
akhilsinghcodes added a commit that referenced this pull request Aug 1, 2026
Add Microsoft's AI Coach analytics: category scorecards, anti-pattern detection, backfill
akhilsinghcodes added a commit that referenced this pull request Aug 1, 2026
Add Microsoft's AI Coach analytics: category scorecards, anti-pattern detection, backfill
akhilsinghcodes added a commit that referenced this pull request Aug 1, 2026
Add Microsoft's AI Coach analytics: category scorecards, anti-pattern detection, backfill
akhilsinghcodes added a commit that referenced this pull request Aug 1, 2026
Add Microsoft's AI Coach analytics: category scorecards, anti-pattern detection, backfill
akhilsinghcodes added a commit that referenced this pull request Aug 1, 2026
Add Microsoft's AI Coach analytics: category scorecards, anti-pattern detection, backfill
akhilsinghcodes added a commit that referenced this pull request Aug 1, 2026
Add Microsoft's AI Coach analytics: category scorecards, anti-pattern detection, backfill
akhilsinghcodes added a commit that referenced this pull request Aug 1, 2026
Add Microsoft's AI Coach analytics: category scorecards, anti-pattern detection, backfill
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant