diff --git a/AI_COACH.md b/AI_COACH.md new file mode 100644 index 0000000..4e08f79 --- /dev/null +++ b/AI_COACH.md @@ -0,0 +1,194 @@ +# AI Coach Analytics + +AgentFleet's **Analytics** tab tells you *how well you're using* Claude Code or Codex, not just how much it cost. It parses the agent's own log files on disk (`~/.claude/projects`, `~/.codex`), scopes them to the exact session you ran, and runs them through 45 detection rules ported from [microsoft/AI-Engineering-Coach](https://github.com/microsoft/AI-Engineering-Coach). The result is a 4-category practice scorecard plus a list of concrete anti-patterns with suggestions. + +![AI Coach analytics scorecard](screenshots/AI_Coach_Analytics_Scorecard.png) + +## Why this exists + +Cost tracking (Spend Analytics) answers "what did this session burn?" It says nothing about *how* you prompted, whether you reviewed the agent's output, or whether you're stuck in a loop. AI Coach answers that second question by mining the raw conversation transcripts the CLI already writes to disk — no extra instrumentation, no telemetry sent anywhere. + +## How it works + +1. **Harness detection** — `command` (e.g. `claude`, `codex`) decides which log parser to use. +2. **Log discovery & parsing** — finds `~/.claude/projects//*.jsonl` (or the Codex equivalent) and parses every request/response pair into a `SessionRequest`. +3. **Session-window scoping** — AgentFleet's "session" (one PTY run) and the CLI's own "session" (one conversation log) aren't the same object. AI Coach filters log sessions to the ones whose timestamps fall inside `[AgentFleet session.created_at − 2min, session.ended_at + 2min]`, so each AgentFleet session gets its own slice of history rather than the repo's entire log archive. +4. **Rule engine** — runs all 45 rules (see below) against the scoped requests/sessions, producing a list of `AntiPattern`s, each tagged with a practice-group, severity, occurrence count, description, and suggestion. +5. **Scoring** — anti-patterns are bucketed into 4 scorecards and scored independently (formula below). The overall **practice score** shown is the average of the 4 category scores. +6. **Persistence** — results are written to `session_analytics` (additive table, doesn't touch existing `sessions`/`pty_chunks` data) and served from `GET /api/analytics/sessions/:id`. + +Existing historical sessions (analyzed before this feature shipped) can be backfilled: + +```bash +pnpm --filter @agents_fleet/server exec tsx scripts/backfill-analytics.ts +``` + +Safe to re-run — it skips sessions that already have analytics and upserts on conflict. + +## Scoring formula + +Each of the 4 scorecards starts at 100 and loses points per detected anti-pattern in that category, weighted by severity: + +| Severity | Penalty | +|---|---| +| high | 12 | +| medium | 7 | +| low | 3 | + +``` +penalty = sum of severity-weights for all anti-patterns in the group +maxPenalty = (number of detectors in the group) × 12 +score = max(0, round(100 × (1 − penalty / maxPenalty))) +``` + +This mirrors the upstream `analyzer-patterns.ts` scoring logic. Week-over-week/month-over-month trend lines from the original project were intentionally **not** ported — there's no rolling history feature in AgentFleet yet, so `wowPct`/`momPct` are always `0`. + +## The 4 practice categories + +### Prompt Quality (16 rules) +How well-formed and well-contextualized your prompts are. + +| Rule | Severity | What it flags | +|---|---|---| +| Lazy Prompting | medium | Short, low-effort prompts that make the agent guess intent | +| Context Engineering Gaps | medium | Missing `AGENTS.md`/`SKILL.md`/MCP tools/file refs/`.instructions.md` | +| Missing File Context | medium | Requests with no `#file` references where they'd help | +| Excessive File Context | medium | Overloading every prompt with too much file context | +| Repeated Prompts | medium | Near-duplicate prompts, signaling the agent didn't understand the first time | +| Frustration Signals | medium | Excessive punctuation / escalating tone — usually means the approach isn't working | +| Caps Lock Rage | medium | ALL-CAPS messages | +| Hostile Language | medium | Profanity/hostile phrasing toward the agent | +| Instruction Bloat | medium | Overlong, over-specified instructions that bury the actual ask | +| Low Constraint Usage | medium | Rarely specifying constraints (style, scope, format) | +| Unstructured Task Starts | medium | Jumping into implementation without a spec/plan | +| No Spec-Driven Development | medium | Never using spec-first workflows for non-trivial work | +| Agent Mode for Simple Questions | medium | Using full agentic mode for things that are really just questions | +| Low Markdown Output Ratio | medium | Model output rarely uses structured markdown (headers/lists/code blocks) | +| Verbose Model Output | medium | Responses are consistently long-winded relative to the ask | +| Verbose Prompts Without Compression | low | Long prompts with no Headroom/compression in use | + +### Session Hygiene (9 rules) +How you manage session length, timing, and flow. + +| Rule | Severity | What it flags | +|---|---|---| +| Mega Sessions | high | Sessions with 50+ messages — context quality degrades | +| Runaway Agent Loops | high | Agentic requests using 15+ tool calls each, possibly spinning | +| Excessive Cancellations | medium | High rate of canceled/interrupted requests | +| Broken Flow State | medium | Frequent context-switching mid-session | +| Session Drift | medium | Conversation wandering away from the original goal | +| Abandoned Sessions | low | Sessions left open/incomplete without a clear end | +| Late-Night Coding | low | Requests between midnight–5am — correlates with lower quality | +| Weekend Overwork | low | Heavy weekend usage | +| Slow Responses | low | Requests taking 30s+ (avg), often from overly broad prompts | + +### Code Review (8 rules) +Whether you're actually reviewing what the agent produces. + +| Rule | Severity | What it flags | +|---|---|---| +| Vibe Coding | high | Accepting agent output with no inspection at all | +| YOLO Mode | high | Auto-accepting all edits/commands without prompts | +| Speed Accept (No Review) | high | Accepting suggestions faster than a human could plausibly read them | +| Copy-Paste Blindness | high | Pasting agent output elsewhere without modification or review | +| Auto-Approved Terminal Commands | medium | Terminal commands set to auto-approve | +| Unsandboxed Terminal Execution | medium | Running agent-issued commands outside a devcontainer/sandbox | +| Single-Workspace Tunnel Vision | low | Never exploring other workspaces/repos for context | +| No Language Exploration | low | Sticking to one language/stack without considering alternatives | + +### Tool Mastery (12 rules) +Whether you're using the harness's model/tool features effectively. + +| Rule | Severity | What it flags | +|---|---|---| +| Premium Model Waste | medium | Using a premium model for trivial tasks | +| Auto Model Avoidance | medium | Pinning a top-tier model instead of using auto-routing | +| Model Overreliance | medium | Always defaulting to the same model regardless of task | +| Premium Model for Lookup Questions | medium | Using a heavy model for simple factual lookups | +| Reasoning Effort Overuse | medium | Requesting max reasoning effort on requests that don't need it | +| Prompt Cache Starvation | medium | Prompt structure prevents cache hits, wasting tokens/cost | +| Tool / MCP Bloat | medium | Too many tools/MCP servers registered, diluting tool selection | +| Never Uses Plan Mode | medium | Skipping plan/dry-run mode for non-trivial changes | +| No Custom Instructions | medium | No `CLAUDE.md`/project instructions configured | +| Agentic Without Tools | low | Agentic-mode requests that end up using no tools at all | +| No Skills Usage | low | Never invoking reusable skills | +| No Slash Commands | low | Never using built-in slash commands | + +## Anti-pattern fields + +Each detected anti-pattern returned by the API has: + +```ts +{ + id: string; // e.g. "auto-avoidance" + name: string; // "Auto Model Avoidance" + severity: "high" | "medium" | "low"; + group: PracticeGroup; // "prompt-quality" | "session-hygiene" | "code-review" | "tool-mastery" + occurrences: number; // how many requests/sessions triggered it + description: string; // what was detected, with real numbers filled in + suggestion: string; // what to do about it +} +``` + +## API + +``` +GET /api/analytics/sessions/:id +``` + +```json +{ + "sessionId": "...", + "harness": "claude", + "practiceScore": 92, + "groupScores": [ + { "group": "prompt-quality", "score": 91, "patternCount": 2, "topIssue": "...", "improvements": [] }, + { "group": "session-hygiene", "score": 83, "patternCount": 3, "topIssue": "...", "improvements": [] }, + { "group": "code-review", "score": 100, "patternCount": 0, "topIssue": null, "improvements": ["All checks passing — no anti-patterns detected."] }, + { "group": "tool-mastery", "score": 92, "patternCount": 1, "topIssue": "...", "improvements": [] } + ], + "antiPatterns": [ /* ... */ ], + "requests": [ /* parsed SessionRequest[] */ ], + "createdAt": "..." +} +``` + +Returns `404` if the session hasn't been analyzed yet (unknown harness, or no logs found in the session's time window). + +## Cross-session dashboard (AI Coach Analytics tab) + +While the per-session view above answers "how did this one session go," the **AI Coach Analytics** tab aggregates every analyzed session over a date range you pick: + +``` +GET /api/ai-coach/dashboard?from=YYYY-MM-DD&to=YYYY-MM-DD # avg score, per-category trends, top anti-patterns, daily activity, harness mix +GET /api/ai-coach/patterns?from=...&to=... # hour × weekday request heatmap, calendar, per-repo project stats +GET /api/ai-coach/timeline?from=...&to=... # most recent 200 sessions in range, with title/duration/score/cost +GET /api/ai-coach/sdlc?from=...&to=... # work-type split (bug fix / feature / refactor / docs / config / code review / other) per request, overall and per-repo +``` + +All four routes join `session_analytics` with `sessions` (and the `session_summary` artifact, for session titles), scoped to `s.created_at BETWEEN from AND to`, and exclude bare `zsh`/`bash` shell sessions. `workType` is a per-request classification already present on the parsed `SessionRequest` from the rule engine — no extra parsing pass needed. + +## Privacy & data flow + +Everything runs locally: +- No network calls — rules run against on-disk JSONL log files you already have. +- Parsed requests, anti-patterns, and scores are stored in your local SQLite DB (`data/agents_fleet.sqlite`), same as the rest of AgentFleet's data. +- Deleting a session also deletes its `session_analytics` row. + +## Implementation + +| Piece | File | +|---|---| +| Parsers, rule engine, 45 rule + 10 metric definitions | [packages/ai-coach-analytics](packages/ai-coach-analytics) | +| Harness detection, session-window scoping, group scoring | [apps/server/src/analytics-adapter.ts](apps/server/src/analytics-adapter.ts) | +| Persists analysis results into SQLite | [apps/server/src/analytics-worker.ts](apps/server/src/analytics-worker.ts) | +| `session_analytics` table | [apps/server/src/db.ts](apps/server/src/db.ts) | +| API route | [apps/server/src/routes/analytics.ts](apps/server/src/routes/analytics.ts) | +| Analytics tab UI (per-session) | [apps/web/src/Analytics.tsx](apps/web/src/Analytics.tsx) | +| Cross-session dashboard API | [apps/server/src/routes/aiCoachAnalytics.ts](apps/server/src/routes/aiCoachAnalytics.ts) | +| Cross-session dashboard UI | [apps/web/src/AiCoachAnalytics.tsx](apps/web/src/AiCoachAnalytics.tsx) | +| Historical backfill script | [apps/server/scripts/backfill-analytics.ts](apps/server/scripts/backfill-analytics.ts) | + +## Credits + +Rule definitions, scoring methodology, and the original concept are from [microsoft/AI-Engineering-Coach](https://github.com/microsoft/AI-Engineering-Coach) (VS Code extension). AgentFleet ports the core parsing/rule-engine logic into a standalone package and re-implements the UI as a web tab instead of a VS Code webview. See [NOTICE](NOTICE) for attribution. diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index c3dbc41..b1b0598 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -19,12 +19,16 @@ graph TD S --> SDK[Claude SDK runner Anthropic] S --> LLM[LiteLLM chat runner proxy backed] S --> REPO[Local repository] + S --> AC[ai-coach-analytics package] PTY -->|runs in cwd| REPO PTY -->|PTY output stream| S S -->|Anthropic SDK messages.create| SDK S -->|OpenAI compatible chat| LLM S -->|tool run_command in repo cwd| REPO + S -->|on session stop, parses logs| AC + AC -->|reads| LOGS[(~/.claude/projects, ~/.codex)] + AC -->|anti-patterns + scores| DB ``` ## SQLite schema (MVP) @@ -38,6 +42,7 @@ The server persists everything to a local SQLite DB at `data/agents_fleet.sqlite | `stdin_events` | Audit trail of user input sent to the PTY (bounded/escaped) | Useful for debugging/auditing without corrupting terminal replay | | `session_markers` | Important timestamps like `stop_requested`, `budget_exceeded`, `process_exit` | Supports replay UX (e.g. freezing replay before TUI cleanup) and debugging session lifecycle | | `session_artifacts` | Per-session artifacts keyed by `session_id` | Stores git snapshots plus chat/session artifacts (transcripts, usage snapshots, tool approvals/results) for later inspection/export | +| `session_analytics` | Practice score + anti-patterns + per-category group scores, keyed by `session_id` | Backs the **Analytics** tab — qualitative coaching data derived from parsing the agent's own log files, additive to (and independent from) the cost/budget data already on `sessions` | ## Notes - Interactive agent CLIs (e.g. Claude Code, Codex) render a full-screen TUI via ANSI escape codes. @@ -63,3 +68,11 @@ The server persists everything to a local SQLite DB at `data/agents_fleet.sqlite - Requests are routed through an OpenAI-compatible LiteLLM proxy endpoint configured by `LITELLM_BASE_URL` and `LITELLM_API_KEY`. - The UI supports model selection, chat history, and the same Approve/Reject workflow for tool calls. - Tool outputs and usage are persisted so budget enforcement and replay remain consistent across providers. + +### AI Coach analytics +- On session stop, `analytics-worker.ts` calls into `@agents_fleet/ai-coach-analytics` (a standalone workspace package ported from [microsoft/AI-Engineering-Coach](https://github.com/microsoft/AI-Engineering-Coach)) to analyze the session — see [AI_COACH.md](AI_COACH.md) for the full breakdown. +- The package parses Claude/Codex log files directly off disk (`~/.claude/projects`, `~/.codex`) — no extra instrumentation, all local. +- AgentFleet's "session" (one PTY run) is reconciled against the CLI's own "session" (one conversation log) by matching the log session's timestamps to the AgentFleet session's `[created_at, ended_at]` window, so analytics stay scoped to that one run rather than the repo's whole log history. +- 45 detection rules run against the scoped requests, producing anti-patterns grouped into 4 practice categories (Prompt Quality, Session Hygiene, Code Review, Tool Mastery), each independently scored. +- Results are written to `session_analytics` (wrapped in `.catch()` — an analytics failure never affects session lifecycle, exit codes, or budget enforcement) and served via `GET /api/analytics/sessions/:id` to the **Analytics** tab. +- Historical sessions can be backfilled with `apps/server/scripts/backfill-analytics.ts`. diff --git a/NOTICE b/NOTICE index 66ffff8..3d6441c 100644 --- a/NOTICE +++ b/NOTICE @@ -13,3 +13,10 @@ Licensed under the MIT License. - License: Apache License 2.0 - Usage: Agents Fleet optionally integrates with Headroom as a local context-compression proxy. Headroom is not bundled or redistributed; it is invoked as an external process at runtime. + +### AI Engineering Coach +- Repository: https://github.com/microsoft/AI-Engineering-Coach +- License: MIT License, Copyright (c) Microsoft Corporation. +- Usage: The session log parsers, anti-pattern rule engine, and rule/metric definitions in + `packages/ai-coach-analytics` are ported from AI Engineering Coach's `src/core` module, adapted + from a VS Code extension into a standalone package consumed by Agents Fleet's web UI. diff --git a/README.md b/README.md index df1d4c1..bd0030b 100644 --- a/README.md +++ b/README.md @@ -11,6 +11,13 @@ Local-first “mission control” for AI coding agent CLIs (and any shell comman ![AgentFleet: Stop Runaway AI Agents with Local Mission Control](screenshots/AgentFleet_Local_AI_Mission_Control.png) ## ✨ Recently Shipped +- **AI Coach analytics** (latest) + - New **Analytics** tab: per-session practice scorecards across 4 categories — Prompt Quality, Session Hygiene, Code Review, Tool Mastery + - 45 built-in detection rules (ported from [AI Engineering Coach](https://github.com/microsoft/AI-Engineering-Coach)) flag anti-patterns like runaway agent loops, late-night coding, lazy prompting, auto-model avoidance — each with severity, occurrence count, and a concrete suggestion + - Parses your own `~/.claude/projects` or `~/.codex` logs and scopes them to the exact AgentFleet session window — no duplicate/global stats + - Backfillable: `pnpm --filter @agents_fleet/server exec tsx scripts/backfill-analytics.ts` analyzes any historical stopped sessions that predate this feature + - New **AI Coach Analytics** tab: cross-session dashboard over a date range — avg practice score, per-category trends, top anti-patterns across all sessions, an hour×weekday activity heatmap, a per-repo project breakdown, a session timeline, and an SDLC work-type split (bug fix / feature / refactor / docs / etc.) + - Full breakdown: [AI_COACH.md](AI_COACH.md) - **AI session summary** (latest) - One-click plain-English summary of any session — title, what the agent did, and token/cost breakdown for the summary call - Powered by `gpt-4o-mini` via your LiteLLM proxy — under $0.001 per summary @@ -72,6 +79,32 @@ This repository contains a **working MVP**: ![Local-first architecture](screenshots/Local_Control_for_AI_Agents.png) +**AI Coach analytics — category scorecards + anti-patterns (per-session)** + +![AI Coach analytics scorecard](screenshots/AI_Coach_Analytics_Scorecard.png) + +- Detailed anti-pattern breakdown (severity, occurrences, suggestion per rule) + +![AI Coach analytics detailed anti-patterns](screenshots/AI_Coach_Analytics_Anti_Patterns.png) + +**AI Coach Analytics — cross-session dashboard** + +- Dashboard — avg practice score, per-category trends, top anti-patterns, daily activity, harness mix + +![AI Coach Analytics dashboard](screenshots/AI_Coach_Analytics_Dashboard.png) + +- Patterns — hour×weekday activity heatmap, session calendar, per-repo project breakdown + +![AI Coach Analytics patterns](screenshots/AI_Coach_Analytics_Patterns.png) + +- Timeline — recent sessions with repo, harness, duration, requests, score, and cost + +![AI Coach Analytics timeline](screenshots/AI_Coach_Analytics_Timeline_Wide.png) + +- SDLC — work-type distribution (bug fix / feature / test / config / docs / style / refactor / code review) overall and per-repo + +![AI Coach Analytics SDLC breakdown](screenshots/AI_Coach_Analytics_SDLC.png) + **Create a new session** - Shell session diff --git a/apps/server/package.json b/apps/server/package.json index b986e9d..c215547 100644 --- a/apps/server/package.json +++ b/apps/server/package.json @@ -13,6 +13,7 @@ "test:watch": "vitest" }, "dependencies": { + "@agents_fleet/ai-coach-analytics": "workspace:*", "@agents_fleet/shared": "workspace:*", "@anthropic-ai/sdk": "^0.100.1", "@homebridge/node-pty-prebuilt-multiarch": "^0.13.1", diff --git a/apps/server/scripts/backfill-analytics.ts b/apps/server/scripts/backfill-analytics.ts new file mode 100644 index 0000000..283ec9c --- /dev/null +++ b/apps/server/scripts/backfill-analytics.ts @@ -0,0 +1,63 @@ +/** + * One-off / re-runnable backfill: analyzes existing stopped sessions that + * don't yet have a session_analytics row. Safe to re-run — skips sessions + * that already have analytics, and analyzeCompletedSession() itself upserts. + */ +import { getDb } from "../src/db"; +import { analyzeCompletedSession } from "../src/analytics-worker"; + +interface SessionRow { + id: string; + command: string; + repo_path: string; + status: string; + created_at: string; + ended_at: string | null; +} + +async function main() { + const db = getDb(); + const sessions = db + .prepare( + `SELECT s.id, s.command, s.repo_path, s.status, s.created_at, s.ended_at + FROM sessions s + LEFT JOIN session_analytics sa ON sa.session_id = s.id + WHERE s.status = 'stopped' AND sa.session_id IS NULL`, + ) + .all() as SessionRow[]; + + console.log(`Found ${sessions.length} stopped session(s) without analytics.`); + + let analyzed = 0; + let skipped = 0; + for (const s of sessions) { + try { + const before = db + .prepare("SELECT 1 FROM session_analytics WHERE session_id = ?") + .get(s.id); + await analyzeCompletedSession( + s.id, + s.command, + s.repo_path, + s.created_at, + s.ended_at ?? undefined, + ); + const after = db + .prepare("SELECT 1 FROM session_analytics WHERE session_id = ?") + .get(s.id); + if (!before && after) { + analyzed++; + console.log(` analyzed ${s.id} (${s.command.slice(0, 40)})`); + } else { + skipped++; + } + } catch (err) { + console.error(` failed ${s.id}:`, err); + skipped++; + } + } + + console.log(`Done. analyzed=${analyzed} skipped=${skipped}`); +} + +main(); diff --git a/apps/server/src/analytics-adapter.ts b/apps/server/src/analytics-adapter.ts new file mode 100644 index 0000000..0e415c4 --- /dev/null +++ b/apps/server/src/analytics-adapter.ts @@ -0,0 +1,165 @@ +import os from "node:os"; +import path from "node:path"; +import { + findClaudeDirs, + findCodexDirs, + getDetectorGroupCounts, + parseClaudeSessions, + parseCodexSessions, + registerAllBuiltinMetrics, + registerAllBuiltinRules, + runDetectors, + type AntiPattern, + type GroupScore, + type PracticeGroup, + type Session, + type SessionRequest, +} from "@agents_fleet/ai-coach-analytics"; + +export type AnalyticsHarness = "claude" | "codex"; + +export interface AnalysisResult { + harness: AnalyticsHarness; + requests: SessionRequest[]; + antiPatterns: AntiPattern[]; + practiceScore: number | null; + groupScores: GroupScore[]; +} + +// Mirrors the upstream AI Coach UI's four scorecards (Prompt Quality, Session +// Hygiene, Code Review, Tool Mastery). "context-management" exists as a +// PracticeGroup value but has no detectors defined upstream either. +const SCORECARD_GROUPS: PracticeGroup[] = [ + "prompt-quality", + "session-hygiene", + "code-review", + "tool-mastery", +]; +const SEVERITY_PENALTY: Record = { + high: 12, + medium: 7, + low: 3, +}; + +function computeGroupScores( + antiPatterns: AntiPattern[], + skipIdeDetectors: boolean, +): GroupScore[] { + const detectorCounts = getDetectorGroupCounts(skipIdeDetectors); + return SCORECARD_GROUPS.map((group) => { + const gPatterns = antiPatterns.filter((p) => p.group === group); + const maxDetectors = detectorCounts[group] || 8; + const penalty = gPatterns.reduce( + (sum, p) => sum + (SEVERITY_PENALTY[p.severity] ?? 5), + 0, + ); + const maxPenalty = maxDetectors * 12; + const score = Math.max(0, Math.round(100 * (1 - penalty / maxPenalty))); + const topIssue = gPatterns[0]?.suggestion ?? null; + const improvements = + gPatterns.length === 0 ? ["All checks passing — no anti-patterns detected."] : []; + return { group, score, wowPct: 0, momPct: 0, topIssue, improvements, patternCount: gPatterns.length }; + }); +} + +function detectHarness(command: string): AnalyticsHarness | null { + const cmd = command.trim().toLowerCase(); + if (cmd.includes("claude")) return "claude"; + if (cmd.includes("codex")) return "codex"; + return null; +} + +// Claude Code / Codex encode a workspace's absolute path into its log directory +// name by replacing every non-alphanumeric character (path separators, dots, +// underscores) with a dash, e.g. /Users/x/my_repo -> -Users-x-my-repo. +function encodeRepoPath(repoPath: string): string { + return repoPath.replace(/[^a-zA-Z0-9]+/g, "-"); +} + +function sessionsForRepo( + groups: { sessions: Session[]; workspaceId: string; workspaceName: string }[], + repoPath: string, +): Session[] { + const encoded = encodeRepoPath(repoPath); + const matches = groups.filter( + (g) => g.workspaceId.endsWith(encoded) || g.workspaceName === repoPath, + ); + return matches.flatMap((g) => g.sessions); +} + +// AgentFleet's "session" is a single PTY process run; the AI Coach "Session" +// is a distinct Claude/Codex conversation log file. They aren't the same +// object, so we scope down to log sessions whose own creation time falls +// inside the AgentFleet run's [started, ended] window (with slack for +// startup/flush latency) rather than returning the whole repo's history. +const WINDOW_SLACK_MS = 2 * 60 * 1000; + +function sessionsInWindow( + sessions: Session[], + startedAt?: string, + endedAt?: string, +): Session[] { + if (!startedAt) return sessions; + const startMs = Date.parse(startedAt) - WINDOW_SLACK_MS; + const endMs = (endedAt ? Date.parse(endedAt) : Date.now()) + WINDOW_SLACK_MS; + return sessions.filter((s) => { + const t = s.creationDate ?? s.lastMessageDate; + return t != null && t >= startMs && t <= endMs; + }); +} + +let rulesRegistered = false; +function ensureRulesRegistered(): void { + if (rulesRegistered) return; + registerAllBuiltinMetrics(); + registerAllBuiltinRules(); + rulesRegistered = true; +} + +export function analyzeSession( + command: string, + repoPath: string, + startedAt?: string, + endedAt?: string, +): AnalysisResult | null { + const harness = detectHarness(command); + if (!harness) return null; + + ensureRulesRegistered(); + + let sessions: Session[]; + if (harness === "claude") { + const dirs = findClaudeDirs(); + if (dirs.length === 0) return null; + const groups = dirs.flatMap((dir) => parseClaudeSessions(dir)); + sessions = sessionsForRepo(groups, repoPath); + } else { + const dirs = findCodexDirs(); + if (dirs.length === 0) return null; + sessions = dirs + .flatMap((dir) => parseCodexSessions(dir)) + .filter((s) => s.workspaceName === repoPath || s.location === repoPath); + } + + sessions = sessionsInWindow(sessions, startedAt, endedAt); + + if (sessions.length === 0) return null; + + const requests = sessions.flatMap((s) => s.requests); + const skipIdeDetectors = true; + const antiPatterns = runDetectors(requests, sessions, skipIdeDetectors); + const groupScores = computeGroupScores(antiPatterns, skipIdeDetectors); + + const practiceScore = + requests.length > 0 + ? Math.round(groupScores.reduce((sum, g) => sum + g.score, 0) / groupScores.length) + : null; + + return { harness, requests, antiPatterns, practiceScore, groupScores }; +} + +// Re-exported for callers that need to resolve log dirs without running analysis. +export const HOME_DIR = os.homedir(); +export function defaultClaudeProjectsDir(): string { + return path.join(HOME_DIR, ".claude", "projects"); +} diff --git a/apps/server/src/analytics-worker.ts b/apps/server/src/analytics-worker.ts new file mode 100644 index 0000000..a55dd8f --- /dev/null +++ b/apps/server/src/analytics-worker.ts @@ -0,0 +1,37 @@ +import crypto from "node:crypto"; +import { getDb } from "./db"; +import { analyzeSession } from "./analytics-adapter"; + +export async function analyzeCompletedSession( + sessionId: string, + command: string, + repoPath: string, + startedAt?: string, + endedAt?: string, +): Promise { + const result = analyzeSession(command, repoPath, startedAt, endedAt); + if (!result) return; + + const db = getDb(); + db.prepare( + `INSERT INTO session_analytics ( + id, session_id, harness, parsed_requests, practice_score, anti_patterns, group_scores, created_at + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?) + ON CONFLICT(session_id) DO UPDATE SET + harness = excluded.harness, + parsed_requests = excluded.parsed_requests, + practice_score = excluded.practice_score, + anti_patterns = excluded.anti_patterns, + group_scores = excluded.group_scores, + created_at = excluded.created_at`, + ).run( + crypto.randomUUID(), + sessionId, + result.harness, + JSON.stringify(result.requests), + result.practiceScore, + JSON.stringify(result.antiPatterns), + JSON.stringify(result.groupScores), + new Date().toISOString(), + ); +} diff --git a/apps/server/src/app.ts b/apps/server/src/app.ts index e681c54..4151a34 100644 --- a/apps/server/src/app.ts +++ b/apps/server/src/app.ts @@ -6,6 +6,8 @@ import { sessionsRouter } from "./routes/sessions"; import { claudeSdkRouter } from "./routes/claudeSdk"; import { liteLlmRouter } from "./routes/litellm"; import { dashboardRouter } from "./routes/dashboard"; +import { analyticsRouter } from "./routes/analytics"; +import { aiCoachAnalyticsRouter } from "./routes/aiCoachAnalytics"; import { getValidModelIds } from "./litellm"; export type AgentsFleetServer = { @@ -37,6 +39,8 @@ export function createApp(): AgentsFleetServer { app.use("/api", claudeSdkRouter(processManager)); app.use("/api", liteLlmRouter(processManager)); app.use("/api", dashboardRouter()); + app.use("/api", analyticsRouter()); + app.use("/api", aiCoachAnalyticsRouter()); return { app, hub, processManager }; } diff --git a/apps/server/src/db.ts b/apps/server/src/db.ts index 88cc8ca..68c7927 100644 --- a/apps/server/src/db.ts +++ b/apps/server/src/db.ts @@ -90,6 +90,27 @@ export function getDb(): Db { CREATE INDEX IF NOT EXISTS idx_session_artifacts_session_ts ON session_artifacts(session_id, timestamp); + + -- Post-hoc qualitative analysis (practice score, anti-patterns) derived from + -- parsing the agent's own log files (e.g. ~/.claude/projects, ~/.codex). + -- Additive only: never read/written by existing budget or replay features. + CREATE TABLE IF NOT EXISTS session_analytics ( + id TEXT PRIMARY KEY, + session_id TEXT NOT NULL UNIQUE, + harness TEXT NOT NULL, + parsed_requests TEXT NOT NULL, + practice_score REAL NULL, + anti_patterns TEXT NOT NULL, + group_scores TEXT NULL, + created_at TEXT NOT NULL, + FOREIGN KEY(session_id) REFERENCES sessions(id) + ); + + CREATE INDEX IF NOT EXISTS idx_session_analytics_session + ON session_analytics(session_id); + + CREATE INDEX IF NOT EXISTS idx_session_analytics_harness + ON session_analytics(harness); `); // Migrations (sessions columns) @@ -120,6 +141,7 @@ export function getDb(): Db { "budget_exceeded_at TEXT NULL", ); ensureColumn(db, "sessions", "stop_reason", "stop_reason TEXT NULL"); + ensureColumn(db, "session_analytics", "group_scores", "group_scores TEXT NULL"); dbSingleton = db; return db; diff --git a/apps/server/src/processManager.ts b/apps/server/src/processManager.ts index 3bf91fe..76fb6d9 100644 --- a/apps/server/src/processManager.ts +++ b/apps/server/src/processManager.ts @@ -4,6 +4,7 @@ import type { Session, SessionStatus } from "@agents_fleet/shared"; import pty, { type IPty } from "@homebridge/node-pty-prebuilt-multiarch"; import { getDb } from "./db"; import { computeCostUsd, estimateTokens } from "./budget"; +import { analyzeCompletedSession } from "./analytics-worker"; import stripAnsi from "strip-ansi"; import type { SessionWsHub } from "./ws"; import { @@ -852,6 +853,15 @@ export class ProcessManager { } if (updated) this.hub.broadcastSession(updated); this.running.delete(args.sessionId); + // Best-effort post-hoc analytics (practice score, anti-patterns). + // Never allowed to affect session lifecycle/state, hence the .catch(). + analyzeCompletedSession( + args.sessionId, + args.command, + args.repoPath, + current?.created_at, + endedAt, + ).catch((err) => console.error("[analytics] analysis failed:", err)); })(); }); } diff --git a/apps/server/src/routes/aiCoachAnalytics.ts b/apps/server/src/routes/aiCoachAnalytics.ts new file mode 100644 index 0000000..ffb4b28 --- /dev/null +++ b/apps/server/src/routes/aiCoachAnalytics.ts @@ -0,0 +1,322 @@ +import type { Request, Response, Router } from "express"; +import { Router as createRouter } from "express"; +import { getDb } from "../db"; + +function jsonError(res: Response, status: number, message: string) { + res.status(status).json({ error: { message } }); +} + +function parseDateRange(req: Request): { from: string; to: string } | null { + const { from, to } = req.query; + if (typeof from !== "string" || typeof to !== "string") return null; + const fromDate = new Date(from); + const toDate = new Date(to); + if (Number.isNaN(fromDate.getTime()) || Number.isNaN(toDate.getTime())) return null; + fromDate.setUTCHours(0, 0, 0, 0); + toDate.setUTCHours(23, 59, 59, 999); + return { from: fromDate.toISOString(), to: toDate.toISOString() }; +} + +interface AnalyticsRow { + session_id: string; + harness: string; + practice_score: number | null; + anti_patterns: string; + group_scores: string | null; + parsed_requests: string; + created_at: string; + // joined from sessions + s_created_at: string; + s_ended_at: string | null; + s_repo_path: string; + s_command: string; + s_estimated_cost_usd: number; + summary_content: string | null; +} + +const JOIN_QUERY = ` + SELECT + sa.session_id, + sa.harness, + sa.practice_score, + sa.anti_patterns, + sa.group_scores, + sa.parsed_requests, + sa.created_at, + s.created_at AS s_created_at, + s.ended_at AS s_ended_at, + s.repo_path AS s_repo_path, + s.command AS s_command, + s.estimated_cost_usd AS s_estimated_cost_usd, + art.content AS summary_content + FROM session_analytics sa + JOIN sessions s ON s.id = sa.session_id + LEFT JOIN session_artifacts art + ON art.session_id = sa.session_id AND art.kind = 'session_summary' + WHERE s.created_at BETWEEN ? AND ? + AND s.command NOT IN ('zsh', 'bash') +`; + +export function aiCoachAnalyticsRouter(): Router { + const router = createRouter(); + + // ── Dashboard ─────────────────────────────────────────────────────────────── + router.get("/ai-coach/dashboard", (req: Request, res: Response) => { + const range = parseDateRange(req); + if (!range) return jsonError(res, 400, "from and to query params required (YYYY-MM-DD)"); + + const db = getDb(); + const rows = db.prepare(JOIN_QUERY).all(range.from, range.to) as AnalyticsRow[]; + + if (rows.length === 0) { + return res.json({ + sessionCount: 0, + avgPracticeScore: null, + groupAverages: [], + topAntiPatterns: [], + dailyActivity: [], + harnessBreakdown: [], + }); + } + + // Overall avg practice score + const scored = rows.filter((r) => r.practice_score != null); + const avgPracticeScore = + scored.length > 0 + ? Math.round(scored.reduce((s, r) => s + (r.practice_score ?? 0), 0) / scored.length) + : null; + + // Per-group averages + const groupSums: Record = {}; + for (const row of rows) { + const gs: { group: string; score: number }[] = row.group_scores + ? JSON.parse(row.group_scores) + : []; + for (const g of gs) { + if (!groupSums[g.group]) groupSums[g.group] = { sum: 0, count: 0 }; + groupSums[g.group].sum += g.score; + groupSums[g.group].count += 1; + } + } + const groupAverages = Object.entries(groupSums).map(([group, { sum, count }]) => ({ + group, + avgScore: Math.round(sum / count), + })); + + // Top anti-patterns by total occurrences + const patternTotals: Record = {}; + for (const row of rows) { + const patterns: { id: string; name: string; group: string; severity: string; occurrences: number }[] = + JSON.parse(row.anti_patterns); + for (const p of patterns) { + if (!patternTotals[p.id]) { + patternTotals[p.id] = { id: p.id, name: p.name, group: p.group, severity: p.severity, totalOccurrences: 0, sessionCount: 0 }; + } + patternTotals[p.id].totalOccurrences += p.occurrences; + patternTotals[p.id].sessionCount += 1; + } + } + const topAntiPatterns = Object.values(patternTotals) + .sort((a, b) => b.sessionCount - a.sessionCount || b.totalOccurrences - a.totalOccurrences) + .slice(0, 10); + + // Daily activity: sessions per day + avg score + const dailyMap: Record = {}; + for (const row of rows) { + const date = row.s_created_at.slice(0, 10); + if (!dailyMap[date]) dailyMap[date] = { date, sessionCount: 0, scoreSum: 0, scoreCount: 0 }; + dailyMap[date].sessionCount += 1; + if (row.practice_score != null) { + dailyMap[date].scoreSum += row.practice_score; + dailyMap[date].scoreCount += 1; + } + } + const dailyActivity = Object.values(dailyMap) + .sort((a, b) => a.date.localeCompare(b.date)) + .map(({ date, sessionCount, scoreSum, scoreCount }) => ({ + date, + sessionCount, + avgScore: scoreCount > 0 ? Math.round(scoreSum / scoreCount) : null, + })); + + // Harness breakdown + const harnessMap: Record = {}; + for (const row of rows) { + if (!harnessMap[row.harness]) harnessMap[row.harness] = { harness: row.harness, count: 0, scoreSum: 0, scoreCount: 0 }; + harnessMap[row.harness].count += 1; + if (row.practice_score != null) { + harnessMap[row.harness].scoreSum += row.practice_score; + harnessMap[row.harness].scoreCount += 1; + } + } + const harnessBreakdown = Object.values(harnessMap).map(({ harness, count, scoreSum, scoreCount }) => ({ + harness, + count, + avgScore: scoreCount > 0 ? Math.round(scoreSum / scoreCount) : null, + })); + + res.json({ + sessionCount: rows.length, + avgPracticeScore, + groupAverages, + topAntiPatterns, + dailyActivity, + harnessBreakdown, + }); + }); + + // ── Patterns ──────────────────────────────────────────────────────────────── + router.get("/ai-coach/patterns", (req: Request, res: Response) => { + const range = parseDateRange(req); + if (!range) return jsonError(res, 400, "from and to query params required (YYYY-MM-DD)"); + + const db = getDb(); + const rows = db.prepare(JOIN_QUERY).all(range.from, range.to) as AnalyticsRow[]; + + // Heatmap: 7 days × 24 hours grid of request counts + // Initialize grid + const heatmap: number[][] = Array.from({ length: 7 }, () => new Array(24).fill(0)); + // Calendar: date → session count + const calendarMap: Record = {}; + // Projects: repoPath → stats + const projectMap: Record }> = {}; + + for (const row of rows) { + // Calendar + const date = row.s_created_at.slice(0, 10); + calendarMap[date] = (calendarMap[date] ?? 0) + 1; + + // Projects + const rp = row.s_repo_path; + if (!projectMap[rp]) projectMap[rp] = { repoPath: rp, sessionCount: 0, requestCount: 0, scoreSum: 0, scoreCount: 0, models: new Set() }; + projectMap[rp].sessionCount += 1; + if (row.practice_score != null) { + projectMap[rp].scoreSum += row.practice_score; + projectMap[rp].scoreCount += 1; + } + + // Heatmap from parsed_requests timestamps + let requests: { timestamp?: number; modelId?: string }[] = []; + try { requests = JSON.parse(row.parsed_requests); } catch { continue; } + projectMap[rp].requestCount += requests.length; + for (const r of requests) { + if (r.modelId) projectMap[rp].models.add(r.modelId); + if (r.timestamp && r.timestamp > 0) { + const d = new Date(r.timestamp); + const dow = d.getUTCDay(); // 0=Sun + const hour = d.getUTCHours(); + heatmap[dow][hour] += 1; + } + } + } + + // Calendar array + const calendar = Object.entries(calendarMap) + .sort(([a], [b]) => a.localeCompare(b)) + .map(([date, count]) => ({ date, count })); + + // Projects array + const projects = Object.values(projectMap) + .sort((a, b) => b.sessionCount - a.sessionCount) + .map(({ repoPath, sessionCount, requestCount, scoreSum, scoreCount, models }) => ({ + repoPath, + sessionCount, + requestCount, + avgScore: scoreCount > 0 ? Math.round(scoreSum / scoreCount) : null, + models: [...models], + })); + + res.json({ heatmap, calendar, projects }); + }); + + // ── Timeline ──────────────────────────────────────────────────────────────── + router.get("/ai-coach/timeline", (req: Request, res: Response) => { + const range = parseDateRange(req); + if (!range) return jsonError(res, 400, "from and to query params required (YYYY-MM-DD)"); + + const db = getDb(); + const rows = db.prepare(JOIN_QUERY + " ORDER BY s.created_at DESC LIMIT 200").all(range.from, range.to) as AnalyticsRow[]; + + const sessions = rows.map((row) => { + let requestCount = 0; + try { + const reqs = JSON.parse(row.parsed_requests); + requestCount = Array.isArray(reqs) ? reqs.length : 0; + } catch { /* empty */ } + + const startMs = row.s_created_at ? Date.parse(row.s_created_at) : null; + const endMs = row.s_ended_at ? Date.parse(row.s_ended_at) : null; + const durationMs = startMs != null && endMs != null ? endMs - startMs : null; + + let sessionTitle: string | null = null; + if (row.summary_content) { + try { sessionTitle = (JSON.parse(row.summary_content) as { title?: string }).title ?? null; } catch { /* empty */ } + } + + return { + sessionId: row.session_id, + title: sessionTitle, + repoPath: row.s_repo_path, + command: row.s_command, + harness: row.harness, + startedAt: row.s_created_at, + endedAt: row.s_ended_at, + durationMs, + requestCount, + practiceScore: row.practice_score, + estimatedCost: row.s_estimated_cost_usd, + }; + }); + + res.json({ sessions }); + }); + + // ── SDLC ──────────────────────────────────────────────────────────────────── + router.get("/ai-coach/sdlc", (req: Request, res: Response) => { + const range = parseDateRange(req); + if (!range) return jsonError(res, 400, "from and to query params required (YYYY-MM-DD)"); + + const db = getDb(); + const rows = db.prepare(JOIN_QUERY).all(range.from, range.to) as AnalyticsRow[]; + + const workTypeCounts: Record = {}; + const byRepoMap: Record> = {}; + let totalRequests = 0; + + for (const row of rows) { + let requests: { workType?: string }[] = []; + try { requests = JSON.parse(row.parsed_requests); } catch { continue; } + totalRequests += requests.length; + + for (const r of requests) { + const wt = r.workType || "other"; + workTypeCounts[wt] = (workTypeCounts[wt] ?? 0) + 1; + if (!byRepoMap[row.s_repo_path]) byRepoMap[row.s_repo_path] = {}; + byRepoMap[row.s_repo_path][wt] = (byRepoMap[row.s_repo_path][wt] ?? 0) + 1; + } + } + + const workTypePct: Record = {}; + for (const [wt, count] of Object.entries(workTypeCounts)) { + workTypePct[wt] = totalRequests > 0 ? Math.round((count / totalRequests) * 100) : 0; + } + + const byRepo = Object.entries(byRepoMap) + .map(([repoPath, counts]) => ({ repoPath, workTypeCounts: counts })) + .sort((a, b) => { + const aTotal = Object.values(a.workTypeCounts).reduce((s, n) => s + n, 0); + const bTotal = Object.values(b.workTypeCounts).reduce((s, n) => s + n, 0); + return bTotal - aTotal; + }); + + res.json({ + sessionCount: rows.length, + totalRequests, + workTypeCounts, + workTypePct, + byRepo, + }); + }); + + return router; +} diff --git a/apps/server/src/routes/analytics.ts b/apps/server/src/routes/analytics.ts new file mode 100644 index 0000000..34b208a --- /dev/null +++ b/apps/server/src/routes/analytics.ts @@ -0,0 +1,45 @@ +import type { Response, Router } from "express"; +import { Router as createRouter } from "express"; +import { getDb } from "../db"; + +function jsonError(res: Response, status: number, message: string) { + res.status(status).json({ error: { message } }); +} + +interface SessionAnalyticsRow { + id: string; + session_id: string; + harness: string; + parsed_requests: string; + practice_score: number | null; + anti_patterns: string; + group_scores: string | null; + created_at: string; +} + +export function analyticsRouter(): Router { + const router = createRouter(); + + router.get("/analytics/sessions/:id", (req, res) => { + const db = getDb(); + const row = db + .prepare("SELECT * FROM session_analytics WHERE session_id = ?") + .get(req.params.id) as SessionAnalyticsRow | undefined; + + if (!row) { + return jsonError(res, 404, "No analytics available for this session"); + } + + res.json({ + sessionId: row.session_id, + harness: row.harness, + practiceScore: row.practice_score, + antiPatterns: JSON.parse(row.anti_patterns), + groupScores: row.group_scores ? JSON.parse(row.group_scores) : [], + requests: JSON.parse(row.parsed_requests), + createdAt: row.created_at, + }); + }); + + return router; +} diff --git a/apps/server/src/routes/sessions.ts b/apps/server/src/routes/sessions.ts index cfa5209..8fcbb77 100644 --- a/apps/server/src/routes/sessions.ts +++ b/apps/server/src/routes/sessions.ts @@ -467,6 +467,7 @@ Respond with JSON only, no markdown: db.prepare("DELETE FROM stdin_events WHERE session_id = ?").run(id); db.prepare("DELETE FROM session_markers WHERE session_id = ?").run(id); db.prepare("DELETE FROM session_artifacts WHERE session_id = ?").run(id); + db.prepare("DELETE FROM session_analytics WHERE session_id = ?").run(id); db.prepare("DELETE FROM sessions WHERE id = ?").run(id); return res.json({}); diff --git a/apps/web/src/AiCoachAnalytics.tsx b/apps/web/src/AiCoachAnalytics.tsx new file mode 100644 index 0000000..b4d467f --- /dev/null +++ b/apps/web/src/AiCoachAnalytics.tsx @@ -0,0 +1,852 @@ +import { useEffect, useMemo, useState } from "react"; +import { + Alert, + Box, + Button, + Chip, + CircularProgress, + LinearProgress, + Paper, + Stack, + Table, + TableBody, + TableCell, + TableContainer, + TableHead, + TableRow, + TableSortLabel, + Tooltip, + Typography, +} from "@mui/material"; +import { + Bar, + BarChart, + Cell, + Pie, + PieChart, + ResponsiveContainer, + Tooltip as RechartsTooltip, + XAxis, + YAxis, +} from "recharts"; +import { + getAiCoachDashboard, + getAiCoachPatterns, + getAiCoachSdlc, + getAiCoachTimeline, + type AiCoachDashboardData, + type AiCoachPatternsData, + type AiCoachSdlcData, + type AiCoachTimelineSession, + PRACTICE_GROUP_LABELS, + type PracticeGroup, +} from "./api"; + +// ── Date helpers ────────────────────────────────────────────────────────────── + +type Preset = "today" | "this_week" | "last_week" | "month" | "ytd" | "custom"; + +function toDateStr(d: Date) { + return `${d.getFullYear()}-${String(d.getMonth() + 1).padStart(2, "0")}-${String(d.getDate()).padStart(2, "0")}`; +} + +function utcWeekStart(now: Date): Date { + const dow = now.getUTCDay(); + const daysFromMon = dow === 0 ? 6 : dow - 1; + const d = new Date(now); + d.setUTCDate(now.getUTCDate() - daysFromMon); + d.setUTCHours(0, 0, 0, 0); + return d; +} + +function toUTCDateStr(d: Date) { + return `${d.getUTCFullYear()}-${String(d.getUTCMonth() + 1).padStart(2, "0")}-${String(d.getUTCDate()).padStart(2, "0")}`; +} + +function presetRange(preset: Preset): { from: string; to: string } { + const now = new Date(); + const today = toDateStr(now); + if (preset === "today") return { from: today, to: today }; + if (preset === "this_week") { + const ws = utcWeekStart(now); + return { from: toUTCDateStr(ws), to: toUTCDateStr(now) }; + } + if (preset === "last_week") { + const ws = utcWeekStart(now); + const end = new Date(ws); end.setUTCDate(ws.getUTCDate() - 1); + const start = new Date(end); start.setUTCDate(end.getUTCDate() - 6); + return { from: toUTCDateStr(start), to: toUTCDateStr(end) }; + } + if (preset === "month") { + return { from: `${now.getFullYear()}-${String(now.getMonth() + 1).padStart(2, "0")}-01`, to: today }; + } + if (preset === "ytd") return { from: `${now.getFullYear()}-01-01`, to: today }; + return { from: today, to: today }; +} + +// ── Score helpers ───────────────────────────────────────────────────────────── + +function scoreColor(score: number): "success" | "warning" | "error" { + if (score >= 70) return "success"; + if (score >= 40) return "warning"; + return "error"; +} + +function scoreHex(score: number) { + if (score >= 70) return "#16a34a"; + if (score >= 40) return "#d97706"; + return "#dc2626"; +} + +const SEVERITY_COLOR: Record = { + high: "error", + medium: "warning", + low: "default", +}; + +function shortPath(p: string) { + const parts = p.split("/"); + return parts[parts.length - 1] || p; +} + +function fmtDuration(ms: number | null) { + if (ms == null) return "—"; + const s = Math.round(ms / 1000); + if (s < 60) return `${s}s`; + const m = Math.floor(s / 60); + const rem = s % 60; + if (m < 60) return `${m}m ${rem}s`; + return `${Math.floor(m / 60)}h ${m % 60}m`; +} + +const WORK_TYPE_COLORS: Record = { + feature: "#0891b2", + "bug fix": "#dc2626", + refactor: "#7c3aed", + "code review": "#d97706", + docs: "#16a34a", + test: "#0284c7", + style: "#db2777", + config: "#64748b", + other: "#94a3b8", +}; + +// ── Date-range picker ───────────────────────────────────────────────────────── + +function DateRangePicker({ + preset, onPreset, customFrom, customTo, onCustomFrom, onCustomTo, +}: { + preset: Preset; + onPreset: (p: Preset) => void; + customFrom: string; + customTo: string; + onCustomFrom: (v: string) => void; + onCustomTo: (v: string) => void; +}) { + const presets: { key: Preset; label: string }[] = [ + { key: "today", label: "Today" }, + { key: "this_week", label: "This week" }, + { key: "last_week", label: "Last week" }, + { key: "month", label: "This month" }, + { key: "ytd", label: "YTD" }, + { key: "custom", label: "Custom" }, + ]; + + const btnSx = (active: boolean) => ({ + textTransform: "none" as const, + fontSize: 12, + fontWeight: active ? 700 : 400, + color: active ? "#0891b2" : "text.secondary", + borderBottom: "2px solid", + borderColor: active ? "#0891b2" : "transparent", + borderRadius: 0, + px: 1.5, + py: 0.75, + bgcolor: "transparent", + }); + + return ( + + {presets.map(({ key, label }) => ( + + ))} + {preset === "custom" && ( + + onCustomFrom(e.target.value)} + style={{ fontSize: 12, padding: "2px 6px", borderRadius: 4, border: "1px solid #cbd5e1" }} /> + – + onCustomTo(e.target.value)} + style={{ fontSize: 12, padding: "2px 6px", borderRadius: 4, border: "1px solid #cbd5e1" }} /> + + )} + + ); +} + +// ── Sub-tab: Dashboard ──────────────────────────────────────────────────────── + +function DashboardTab({ from, to }: { from: string; to: string }) { + const [data, setData] = useState(null); + const [loading, setLoading] = useState(false); + const [error, setError] = useState(null); + + useEffect(() => { + setLoading(true); + setError(null); + getAiCoachDashboard(from, to) + .then(setData) + .catch((e: Error) => setError(e.message)) + .finally(() => setLoading(false)); + }, [from, to]); + + if (loading) return ; + if (error) return {error}; + if (!data || data.sessionCount === 0) { + return No analyzed sessions in this period.; + } + + const SCORECARD_GROUPS: PracticeGroup[] = ["prompt-quality", "session-hygiene", "code-review", "tool-mastery"]; + const groupMap = new Map(data.groupAverages.map((g) => [g.group, g.avgScore])); + + const chartData = data.dailyActivity.map((d) => ({ + date: d.date.slice(5), // MM-DD + sessions: d.sessionCount, + score: d.avgScore, + })); + + return ( + + {/* Header stats */} + + + Sessions analyzed + {data.sessionCount} + + + Avg practice score + + {data.avgPracticeScore ?? "—"} + + + {data.harnessBreakdown.map((h) => ( + + {h.harness} sessions + {h.count} + {h.avgScore != null && ( + avg {h.avgScore} + )} + + ))} + + + {/* Scorecard group averages */} + + {SCORECARD_GROUPS.map((group) => { + const score = groupMap.get(group) ?? null; + return ( + + {PRACTICE_GROUP_LABELS[group]} + {score ?? "—"} + {score != null && ( + + )} + + ); + })} + + + {/* Daily activity chart */} + {chartData.length > 0 && ( + + Daily Sessions + + + + + + + + + + )} + + {/* Top anti-patterns */} + + Most frequent anti-patterns ({data.topAntiPatterns.length}) + + {data.topAntiPatterns.length === 0 ? ( + No anti-patterns detected in this period. + ) : ( + + {data.topAntiPatterns.map((p) => ( + + + + + {p.name} + + {p.sessionCount} session{p.sessionCount !== 1 ? "s" : ""} · {p.totalOccurrences} occurrences + + + + ))} + + )} + + ); +} + +// ── Sub-tab: Patterns ───────────────────────────────────────────────────────── + +const DAYS = ["Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat"]; + +function Heatmap({ heatmap }: { heatmap: number[][] }) { + const max = Math.max(1, ...heatmap.flatMap((row) => row)); + return ( + + Activity by Hour & Day + + {/* Hour labels */} + + {DAYS.map((d) => ( + + {d} + + ))} + + {/* Grid */} + + {/* Hour axis */} + + {Array.from({ length: 24 }, (_, h) => ( + + {h % 6 === 0 && {h}h} + + ))} + + {heatmap.map((row, dow) => ( + + {row.map((count, hour) => { + const opacity = count === 0 ? 0.06 : 0.15 + (count / max) * 0.85; + return ( + + + + ); + })} + + ))} + + + + ); +} + +function CalendarHeatmap({ calendar }: { calendar: { date: string; count: number }[] }) { + const countByDate = new Map(calendar.map((c) => [c.date, c.count])); + const max = Math.max(1, ...calendar.map((c) => c.count)); + + // Build 12-week rolling window ending today + const today = new Date(); + const weeks: Date[][] = []; + // Find last Sunday + const end = new Date(today); + end.setDate(today.getDate() + (6 - today.getDay())); + const start = new Date(end); + start.setDate(end.getDate() - 7 * 16 + 1); + + for (let w = 0; w < 16; w++) { + const week: Date[] = []; + for (let d = 0; d < 7; d++) { + const date = new Date(start); + date.setDate(start.getDate() + w * 7 + d); + week.push(date); + } + weeks.push(week); + } + + return ( + + Session Calendar (last 16 weeks) + + + {DAYS.map((d) => ( + + {[1, 3, 5].includes(DAYS.indexOf(d)) && ( + {d} + )} + + ))} + + + {weeks.map((week, wi) => { + const monthLabel = week[0].toLocaleString("default", { month: "short" }); + const showMonth = wi === 0 || week[0].getDate() <= 7; + return ( + + + {showMonth && {monthLabel}} + + {week.map((date) => { + const ds = toDateStr(date); + const count = countByDate.get(ds) ?? 0; + const opacity = count === 0 ? 0.06 : 0.2 + (count / max) * 0.8; + const isFuture = date > today; + return ( + + + + ); + })} + + ); + })} + + + + ); +} + +function PatternsTab({ from, to }: { from: string; to: string }) { + const [data, setData] = useState(null); + const [loading, setLoading] = useState(false); + const [error, setError] = useState(null); + + useEffect(() => { + setLoading(true); + setError(null); + getAiCoachPatterns(from, to) + .then(setData) + .catch((e: Error) => setError(e.message)) + .finally(() => setLoading(false)); + }, [from, to]); + + if (loading) return ; + if (error) return {error}; + if (!data) return null; + + return ( + + + + + + + + + + + + Projects + {data.projects.length === 0 ? ( + No project data in this period. + ) : ( + + + + + Repo + Sessions + Requests + Avg score + Models + + + + {data.projects.map((p) => ( + + + + {shortPath(p.repoPath)} + + + {p.sessionCount} + {p.requestCount} + + {p.avgScore != null ? ( + {p.avgScore} + ) : "—"} + + + {p.models.slice(0, 3).map((m) => m.split("-").slice(-2).join("-")).join(", ")} + {p.models.length > 3 && ` +${p.models.length - 3}`} + + + ))} + +
+ + )} + + + + ); +} + +// ── Sub-tab: Timeline ───────────────────────────────────────────────────────── + +type TimelineSortField = "startedAt" | "durationMs" | "requestCount" | "practiceScore" | "estimatedCost"; +type SortDir = "asc" | "desc"; + +function TimelineTab({ from, to }: { from: string; to: string }) { + const [sessions, setSessions] = useState([]); + const [loading, setLoading] = useState(false); + const [error, setError] = useState(null); + const [sort, setSort] = useState<{ field: TimelineSortField; dir: SortDir }>({ field: "startedAt", dir: "desc" }); + + useEffect(() => { + setLoading(true); + setError(null); + getAiCoachTimeline(from, to) + .then(setSessions) + .catch((e: Error) => setError(e.message)) + .finally(() => setLoading(false)); + }, [from, to]); + + const sorted = useMemo(() => { + return [...sessions].sort((a, b) => { + const av = a[sort.field] ?? 0; + const bv = b[sort.field] ?? 0; + const cmp = typeof av === "number" && typeof bv === "number" + ? av - bv + : String(av).localeCompare(String(bv)); + return sort.dir === "asc" ? cmp : -cmp; + }); + }, [sessions, sort]); + + function toggleSort(field: TimelineSortField) { + setSort((prev) => prev.field === field ? { field, dir: prev.dir === "asc" ? "desc" : "asc" } : { field, dir: "desc" }); + } + + const colSx = { fontWeight: 700, fontSize: 12, whiteSpace: "nowrap" as const }; + + if (loading) return ; + if (error) return {error}; + if (sessions.length === 0) { + return No analyzed sessions in this period.; + } + + return ( + + + {sessions.length} session{sessions.length !== 1 ? "s" : ""} analyzed + + + + + + Session + Repo + Harness + + toggleSort("startedAt")}> + Started + + + + toggleSort("durationMs")}> + Duration + + + + toggleSort("requestCount")}> + Requests + + + + toggleSort("practiceScore")}> + Score + + + + toggleSort("estimatedCost")}> + Cost + + + + + + {sorted.map((s) => ( + + + + + {s.title ?? s.command.replace(/^\[.*?\]:/, "").slice(0, 40)} + + + + + + {shortPath(s.repoPath)} + + + + + + + {new Date(s.startedAt).toLocaleString(undefined, { month: "short", day: "numeric", hour: "2-digit", minute: "2-digit" })} + + {fmtDuration(s.durationMs)} + {s.requestCount} + + {s.practiceScore != null ? ( + {s.practiceScore} + ) : —} + + + {s.estimatedCost > 0 ? `$${s.estimatedCost.toFixed(4)}` : "—"} + + + ))} + +
+ + + ); +} + +// ── Sub-tab: SDLC ───────────────────────────────────────────────────────────── + +function SdlcTab({ from, to }: { from: string; to: string }) { + const [data, setData] = useState(null); + const [loading, setLoading] = useState(false); + const [error, setError] = useState(null); + + useEffect(() => { + setLoading(true); + setError(null); + getAiCoachSdlc(from, to) + .then(setData) + .catch((e: Error) => setError(e.message)) + .finally(() => setLoading(false)); + }, [from, to]); + + if (loading) return ; + if (error) return {error}; + if (!data || data.totalRequests === 0) { + return No SDLC data in this period.; + } + + const pieData = Object.entries(data.workTypeCounts) + .sort((a, b) => b[1] - a[1]) + .map(([name, value]) => ({ name, value, pct: data.workTypePct[name] ?? 0 })); + + return ( + + + {/* Donut chart */} + + Work type distribution + + {data.totalRequests.toLocaleString()} requests across {data.sessionCount} sessions + + + + + + {pieData.map((entry) => ( + + ))} + + [ + `${value.toLocaleString()} requests (${data.workTypePct[name] ?? 0}%)`, + (name as string).replace(/\b\w/g, (c) => c.toUpperCase()), + ]} + contentStyle={{ fontSize: 12, borderRadius: 6 }} + /> + + + {/* Centre label */} + + {data.totalRequests.toLocaleString()} + requests + + + {/* Colour legend */} + + {pieData.map((entry) => ( + + + {entry.name} {entry.pct}% + + ))} + + + + {/* Summary table */} + + By work type + + + + + Work type + Requests + Share + + + + {pieData.map((row) => ( + + + + + {row.name} + + + {row.value.toLocaleString()} + {row.pct}% + + ))} + +
+ + + + + {/* Per-repo breakdown */} + {data.byRepo.length > 0 && ( + + By repo + + + + + Repo + {pieData.slice(0, 6).map((wt) => ( + + + + {wt.name} + + + ))} + + + + {data.byRepo.slice(0, 15).map((repo) => ( + + + + {shortPath(repo.repoPath)} + + + {pieData.slice(0, 6).map((wt) => ( + + {repo.workTypeCounts[wt.name] ?? "—"} + + ))} + + ))} + +
+ + + )} + + ); +} + +// ── Root component ──────────────────────────────────────────────────────────── + +type CoachTab = "dashboard" | "patterns" | "timeline" | "sdlc"; + +const COACH_TABS: { key: CoachTab; label: string }[] = [ + { key: "dashboard", label: "Dashboard" }, + { key: "patterns", label: "Patterns" }, + { key: "timeline", label: "Timeline" }, + { key: "sdlc", label: "SDLC" }, +]; + +export function AiCoachAnalyticsContent() { + const [tab, setTab] = useState("dashboard"); + const [preset, setPreset] = useState("this_week"); + const [customFrom, setCustomFrom] = useState(() => toDateStr(new Date())); + const [customTo, setCustomTo] = useState(() => toDateStr(new Date())); + + const { from, to } = useMemo(() => { + if (preset === "custom") return { from: customFrom, to: customTo }; + return presetRange(preset); + }, [preset, customFrom, customTo]); + + const tabBtnSx = (active: boolean) => ({ + textTransform: "none" as const, + fontSize: 13, + fontWeight: active ? 700 : 400, + color: active ? "primary.main" : "text.secondary", + borderBottom: "2px solid", + borderColor: active ? "primary.main" : "transparent", + borderRadius: 0, + px: 2, + py: 1, + bgcolor: "transparent", + }); + + return ( + + {/* Sub-tab bar + date range */} + + + {COACH_TABS.map(({ key, label }) => ( + + ))} + + + + + + + {/* Content */} + + {tab === "dashboard" && } + {tab === "patterns" && } + {tab === "timeline" && } + {tab === "sdlc" && } + + + ); +} diff --git a/apps/web/src/Analytics.tsx b/apps/web/src/Analytics.tsx new file mode 100644 index 0000000..cc93ec8 --- /dev/null +++ b/apps/web/src/Analytics.tsx @@ -0,0 +1,179 @@ +import { useEffect, useState } from "react"; +import { + getSessionAnalytics, + PRACTICE_GROUP_LABELS, + type PracticeGroup, + type SessionAnalytics, +} from "./api"; +import { + Alert, + Box, + Chip, + CircularProgress, + LinearProgress, + Paper, + Stack, + Typography, +} from "@mui/material"; + +const SEVERITY_COLOR: Record = { + high: "error", + medium: "warning", + low: "default", +}; + +const SCORECARD_GROUPS: PracticeGroup[] = [ + "prompt-quality", + "session-hygiene", + "code-review", + "tool-mastery", +]; + +function scoreColor(score: number): "success" | "warning" | "error" { + if (score >= 70) return "success"; + if (score >= 40) return "warning"; + return "error"; +} + +export function AnalyticsContent({ sessionId }: { sessionId: string | null }) { + const [data, setData] = useState(null); + const [loading, setLoading] = useState(false); + const [error, setError] = useState(null); + + useEffect(() => { + if (!sessionId) { + setData(null); + return; + } + setLoading(true); + setError(null); + getSessionAnalytics(sessionId) + .then(setData) + .catch((err) => setError(String(err))) + .finally(() => setLoading(false)); + }, [sessionId]); + + if (!sessionId) { + return ( + + Select a session to see its analytics. + + ); + } + + if (loading) { + return ( + + + + ); + } + + if (error) { + return ( + + {error} + + ); + } + + if (!data) { + return ( + + Not analyzed yet — analytics are generated for Claude and Codex + sessions after they finish. + + ); + } + + const scoresByGroup = new Map(data.groupScores.map((g) => [g.group, g])); + const groupOrder: Record = { + "prompt-quality": 0, + "session-hygiene": 1, + "code-review": 2, + "tool-mastery": 3, + }; + const sortedPatterns = [...data.antiPatterns].sort( + (a, b) => + (groupOrder[a.group] ?? 9) - (groupOrder[b.group] ?? 9) || + b.occurrences - a.occurrences, + ); + + return ( + + + {SCORECARD_GROUPS.map((group) => { + const g = scoresByGroup.get(group); + const score = g?.score ?? 100; + return ( + + + {PRACTICE_GROUP_LABELS[group]} + + + {score} + + + + {g && g.patternCount > 0 + ? `${g.patternCount} issue(s) detected` + : "All checks passing"} + + + ); + })} + + + Harness + + + {data.harness} + + + + + + Detected anti-patterns ({data.antiPatterns.length}) + + {sortedPatterns.length === 0 ? ( + + No anti-patterns detected. + + ) : ( + + {sortedPatterns.map((p) => ( + + + + + + {p.name} + + + ({p.occurrences} occurrences) + + + + {p.description} + + {p.suggestion && ( + + {p.suggestion} + + )} + + ))} + + )} + + ); +} diff --git a/apps/web/src/App.tsx b/apps/web/src/App.tsx index 6bd2510..504aa56 100644 --- a/apps/web/src/App.tsx +++ b/apps/web/src/App.tsx @@ -2,6 +2,8 @@ import type { Session } from "@agents_fleet/shared"; import React, { useEffect, useMemo, useRef, useState } from "react"; import { createSession, deleteSession, listSessions, stopSession } from "./api"; import ClaudeSdkChat from "./ClaudeSdkChat"; +import { AnalyticsContent } from "./Analytics"; +import { AiCoachAnalyticsContent } from "./AiCoachAnalytics"; import { DashboardContent } from "./Dashboard"; import HeadroomChat from "./HeadroomChat"; import LiteLLMChat from "./LiteLLMChat"; @@ -33,7 +35,7 @@ import { // ── Types ───────────────────────────────────────────────────────────────────── -type LeftTab = "shell" | "claude_sdk" | "litellm" | "headroom_chat" | "headroom_shell" | "dashboard"; +type LeftTab = "shell" | "claude_sdk" | "litellm" | "headroom_chat" | "headroom_shell" | "dashboard" | "analytics" | "ai_coach"; type CenterTab = "terminal" | "logs" | "artifacts"; // ── Status badge ────────────────────────────────────────────────────────────── @@ -404,6 +406,8 @@ function MainApp() { { key: "headroom_chat", label: "Headroom Chat" }, { key: "headroom_shell", label: "Headroom Shell" }, { key: "dashboard", label: "Spend Analytics" }, + { key: "analytics", label: "Analytics" }, + { key: "ai_coach", label: "AI Coach Analytics" }, ]; // ── Center-tab config ──────────────────────────────────────────────────────── @@ -418,7 +422,7 @@ function MainApp() { sx={{ height: "100vh", display: "grid", - gridTemplateColumns: (leftTab === "dashboard") ? "1fr" : "1fr 300px", + gridTemplateColumns: (leftTab === "dashboard" || leftTab === "ai_coach") ? "1fr" : "1fr 300px", gridTemplateRows: "1fr", bgcolor: "background.default", overflow: "hidden", @@ -567,6 +571,10 @@ function MainApp() { /> )} + ) : leftTab === "analytics" ? ( + + ) : leftTab === "ai_coach" ? ( + ) : leftTab === "headroom_shell" ? ( /* Headroom Shell tab - identical to Shell tab but with headroom: true */ diff --git a/apps/web/src/api.ts b/apps/web/src/api.ts index 331ed35..212ea6b 100644 --- a/apps/web/src/api.ts +++ b/apps/web/src/api.ts @@ -129,6 +129,172 @@ export async function listSessions(): Promise { return json.sessions; } +export interface SessionAnalyticsAntiPattern { + id: string; + name: string; + severity: "high" | "medium" | "low"; + group: PracticeGroup; + occurrences: number; + description: string; + suggestion: string; +} + +export type PracticeGroup = + | "prompt-quality" + | "session-hygiene" + | "code-review" + | "tool-mastery" + | "context-management"; + +export interface SessionAnalyticsGroupScore { + group: PracticeGroup; + score: number; + topIssue: string | null; + improvements: string[]; + patternCount: number; +} + +export const PRACTICE_GROUP_LABELS: Record = { + "prompt-quality": "Prompt Quality", + "session-hygiene": "Session Hygiene", + "code-review": "Code Review", + "tool-mastery": "Tool Mastery", + "context-management": "Context Management", +}; + +export interface SessionAnalytics { + sessionId: string; + harness: string; + practiceScore: number | null; + antiPatterns: SessionAnalyticsAntiPattern[]; + groupScores: SessionAnalyticsGroupScore[]; + createdAt: string; +} + +export async function getSessionAnalytics( + id: string, +): Promise { + const res = await fetch(`/api/analytics/sessions/${encodeURIComponent(id)}`); + if (res.status === 404) return null; + const json = await parseJson(res); + if (isApiError(json)) throw new Error(json.error.message); + if (!res.ok) throw new Error("Request failed"); + return json; +} + +export interface AiCoachGroupAverage { + group: PracticeGroup; + avgScore: number; +} + +export interface AiCoachTopAntiPattern { + id: string; + name: string; + group: PracticeGroup; + severity: "high" | "medium" | "low"; + totalOccurrences: number; + sessionCount: number; +} + +export interface AiCoachDailyActivity { + date: string; + sessionCount: number; + avgScore: number | null; +} + +export interface AiCoachHarnessBreakdown { + harness: string; + count: number; + avgScore: number | null; +} + +export interface AiCoachDashboardData { + sessionCount: number; + avgPracticeScore: number | null; + groupAverages: AiCoachGroupAverage[]; + topAntiPatterns: AiCoachTopAntiPattern[]; + dailyActivity: AiCoachDailyActivity[]; + harnessBreakdown: AiCoachHarnessBreakdown[]; +} + +export interface AiCoachProject { + repoPath: string; + sessionCount: number; + requestCount: number; + avgScore: number | null; + models: string[]; +} + +export interface AiCoachCalendarDay { + date: string; + count: number; +} + +export interface AiCoachPatternsData { + heatmap: number[][]; + calendar: AiCoachCalendarDay[]; + projects: AiCoachProject[]; +} + +export interface AiCoachTimelineSession { + sessionId: string; + title: string | null; + repoPath: string; + command: string; + harness: string; + startedAt: string; + endedAt: string | null; + durationMs: number | null; + requestCount: number; + practiceScore: number | null; + estimatedCost: number; +} + +export interface AiCoachByRepoSdlc { + repoPath: string; + workTypeCounts: Record; +} + +export interface AiCoachSdlcData { + sessionCount: number; + totalRequests: number; + workTypeCounts: Record; + workTypePct: Record; + byRepo: AiCoachByRepoSdlc[]; +} + +function dateRangeParams(from: string, to: string): string { + return `from=${encodeURIComponent(from)}&to=${encodeURIComponent(to)}`; +} + +export async function getAiCoachDashboard(from: string, to: string): Promise { + const res = await fetch(`/api/ai-coach/dashboard?${dateRangeParams(from, to)}`); + const json = await parseJson(res); + if (isApiError(json)) throw new Error(json.error.message); + return json as AiCoachDashboardData; +} + +export async function getAiCoachPatterns(from: string, to: string): Promise { + const res = await fetch(`/api/ai-coach/patterns?${dateRangeParams(from, to)}`); + const json = await parseJson(res); + if (isApiError(json)) throw new Error(json.error.message); + return json as AiCoachPatternsData; +} + +export async function getAiCoachTimeline(from: string, to: string): Promise { + const res = await fetch(`/api/ai-coach/timeline?${dateRangeParams(from, to)}`); + const json = await parseJson<{ sessions: AiCoachTimelineSession[] } | ApiErrorShape>(res); + if (isApiError(json)) throw new Error(json.error.message); + return (json as { sessions: AiCoachTimelineSession[] }).sessions; +} + +export async function getAiCoachSdlc(from: string, to: string): Promise { + const res = await fetch(`/api/ai-coach/sdlc?${dateRangeParams(from, to)}`); + const json = await parseJson(res); + if (isApiError(json)) throw new Error(json.error.message); + return json as AiCoachSdlcData; +} + export async function getSession(id: string): Promise { const res = await fetch(`/api/sessions/${encodeURIComponent(id)}`); const json = await parseJson<{ session: Session } | ApiErrorShape>(res); diff --git a/packages/ai-coach-analytics/eslint.config.js b/packages/ai-coach-analytics/eslint.config.js new file mode 100644 index 0000000..1eca821 --- /dev/null +++ b/packages/ai-coach-analytics/eslint.config.js @@ -0,0 +1,16 @@ +import js from "@eslint/js"; +import tseslint from "typescript-eslint"; + +export default [ + js.configs.recommended, + ...tseslint.configs.recommended, + { + ignores: ["dist/**", "node_modules/**"], + rules: { + "@typescript-eslint/no-unused-vars": [ + "error", + { argsIgnorePattern: "^_", varsIgnorePattern: "^_" }, + ], + }, + }, +]; diff --git a/packages/ai-coach-analytics/package.json b/packages/ai-coach-analytics/package.json new file mode 100644 index 0000000..6ed5e11 --- /dev/null +++ b/packages/ai-coach-analytics/package.json @@ -0,0 +1,24 @@ +{ + "name": "@agents_fleet/ai-coach-analytics", + "version": "0.0.0", + "private": true, + "type": "module", + "exports": { + ".": "./src/index.ts" + }, + "scripts": { + "typecheck": "tsc -p tsconfig.json --noEmit", + "lint": "eslint ." + }, + "dependencies": { + "zod": "^4.4.3", + "leo-profanity": "^1.9.0" + }, + "devDependencies": { + "typescript": "^5.6.3", + "eslint": "^9.9.1", + "@eslint/js": "^9.9.1", + "typescript-eslint": "^8.4.0", + "@types/node": "^22.13.5" + } +} diff --git a/packages/ai-coach-analytics/src/analyzer-base.ts b/packages/ai-coach-analytics/src/analyzer-base.ts new file mode 100644 index 0000000..5369bb7 --- /dev/null +++ b/packages/ai-coach-analytics/src/analyzer-base.ts @@ -0,0 +1,87 @@ +/*--------------------------------------------------------------------------------------------- + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the MIT License. See LICENSE in the project root for license information. + *--------------------------------------------------------------------------------------------*/ + +/* Base class for analyzer modules -- provides shared filtering logic */ + +import { Session, SessionRequest, DateFilter } from './types'; +import { toDateStr } from './helpers'; + +export class AnalyzerBase { + protected readonly sessions: Session[]; + protected readonly editLocIndex: Map>; + protected readonly requestSessionMap: Map; + + constructor(sessions: Session[], editLocIndex: Map>, sharedMap?: Map) { + this.sessions = sessions; + this.editLocIndex = editLocIndex; + if (sharedMap) { + this.requestSessionMap = sharedMap; + } else { + this.requestSessionMap = AnalyzerBase.buildRequestSessionMap(sessions); + } + } + + static buildRequestSessionMap(sessions: Session[]): Map { + const map = new Map(); + for (const s of sessions) { + for (const r of s.requests) { + map.set(r, s); + } + } + return map; + } + + protected matchesWorkspaceFilter(session: Session, workspaceId?: string): boolean { + if (!workspaceId) return true; + return session.workspaceId === workspaceId || session.workspaceName === workspaceId; + } + + protected filter(f?: DateFilter): SessionRequest[] { + const reqs: SessionRequest[] = []; + for (const s of this.sessions) { + if (!this.matchesWorkspaceFilter(s, f?.workspaceId)) continue; + if (f?.harness && s.harness !== f.harness) continue; + for (const r of s.requests) { + if (r.timestamp == null || r.timestamp <= 0) continue; + if (f?.fromDate && toDateStr(r.timestamp) < f.fromDate) continue; + if (f?.toDate && toDateStr(r.timestamp) > f.toDate) continue; + reqs.push(r); + } + } + return reqs; + } + + /** Total LoC for a request: code-block lines + agent-mode edit lines. */ + protected requestLoc(r: SessionRequest): number { + let loc = r.aiCode.reduce((s, b) => s + b.loc, 0); + const eMap = this.editLocIndex.get(r.requestId); + if (eMap) for (const v of eMap.values()) loc += v; + return loc; + } + + protected filteredSessions(f?: DateFilter): Session[] { + return this.sessions.filter(s => { + if (!this.matchesWorkspaceFilter(s, f?.workspaceId)) return false; + if (f?.harness && s.harness !== f.harness) return false; + const ts = s.lastMessageDate || s.creationDate; + if (ts == null || ts <= 0) return false; + const d = toDateStr(ts); + if (f?.fromDate && d < f.fromDate) return false; + if (f?.toDate && d > f.toDate) return false; + return true; + }); + } + + /** Ensure the day-key array includes fromDate so fillDayRange starts + * from the filter boundary, keeping all charts aligned on the same x-axis. */ + protected anchorFromDate(keys: string[], f?: DateFilter): string[] { + if (!f?.fromDate || f.fromDate <= '0001-01-01') return keys; + const sorted = keys.length > 0 ? [...keys].sort() : []; + if (sorted.length === 0 || f.fromDate < sorted[0]) { + return [f.fromDate, ...keys]; + } + return keys; + } +} diff --git a/packages/ai-coach-analytics/src/analyzer-insights.ts b/packages/ai-coach-analytics/src/analyzer-insights.ts new file mode 100644 index 0000000..9fe05fb --- /dev/null +++ b/packages/ai-coach-analytics/src/analyzer-insights.ts @@ -0,0 +1,647 @@ +/*--------------------------------------------------------------------------------------------- + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the MIT License. See LICENSE in the project root for license information. + *--------------------------------------------------------------------------------------------*/ + +/* Insights analyzer -- learning velocity, intent classification, spec-driven, + production/review ratio, sustainable pace, prompt maturity, migration readiness */ + +import { + Session, DateFilter, SessionIntent, SESSION_INTENTS, + LearningVelocityData, IntentClassificationData, SpecDrivenData, + ProductionReviewData, SustainablePaceData, PromptMaturityData, + MigrationReadinessData, InsightsData, +} from './types'; +import { toDateStr, isoWeek, classifyWorkType } from './helpers'; +import { + REVIEW_GAP_THRESHOLD_MS, LATE_NIGHT_START, LATE_NIGHT_END, + BURNOUT_STREAK_DAYS, BURNOUT_LATE_NIGHT_RATE, BURNOUT_WEEKEND_RATE, + PROMPT_MATURITY_SAMPLE_SIZE, +} from './constants'; +import { AnalyzerBase } from './analyzer-base'; + +/* ════════════════════════════════════════════════════════════════════ + Intent classification helpers + ════════════════════════════════════════════════════════════════════ */ + +const PLANNING_RE = /\b(plan|architect|design|outline|approach|strategy|scope|breakdown|roadmap|RFC|spec|proposal)\b/i; +const DEBUG_RE = /\b(fix|bug|error|exception|crash|debug|stacktrace|trace|issue|broken|fail|wrong|not working|undefined is not|cannot read|segfault|panic)\b/i; +const REVIEW_RE = /\b(review|explain|understand|what does|how does|walk me through|read|audit|analyze|inspect|clarify|describe)\b/i; +const EXPLORE_RE = /\b(how to|what is|can I|learn|explore|example|tutorial|demo|try|experiment|compare|research|playground)\b/i; + +function classifyIntent(session: Session): SessionIntent { + const scores: Record = { + Planning: 0, Implementation: 0, Debugging: 0, Review: 0, Exploration: 0, + }; + + for (const r of session.requests) { + const msg = r.messageText; + if (r.agentMode.includes('plan') || r.slashCommand === 'plan' || PLANNING_RE.test(msg)) scores.Planning++; + if (DEBUG_RE.test(msg) || r.slashCommand === 'fix') scores.Debugging++; + if (REVIEW_RE.test(msg) || r.slashCommand === 'explain') scores.Review++; + if (EXPLORE_RE.test(msg)) scores.Exploration++; + // Implementation signals: code output, edits, build keywords + if (r.aiCode.length > 0 || r.editedFiles.length > 0) scores.Implementation++; + const wt = r.workType || classifyWorkType(msg); + if (wt === 'feature' || wt === 'refactor' || wt === 'test' || wt === 'config' || wt === 'style') scores.Implementation++; + } + + let best: SessionIntent = 'Implementation'; + let max = 0; + for (const intent of SESSION_INTENTS) { + if (scores[intent] > max) { max = scores[intent]; best = intent; } + } + return best; +} + +/* ════════════════════════════════════════════════════════════════════ + Spec-driven detection helpers + ════════════════════════════════════════════════════════════════════ */ + +const SPEC_FILE_RE = /\.(md|txt|spec|prd|design|plan|rfc|adoc)$/i; +const SPEC_KEYWORD_RE = /\b(spec|requirements?|acceptance criteria|design doc|PRD|RFC|plan file|constraint|must|should|ensure)\b/i; + +function isSpecDriven(session: Session): boolean { + if (session.requests.length === 0) return false; + const first = session.requests[0]; + // References a spec-like file + if (first.referencedFiles.some(f => SPEC_FILE_RE.test(f))) return true; + // First prompt has structured format + const msg = first.messageText; + if (SPEC_KEYWORD_RE.test(msg)) return true; + if (/^[-*]\s/m.test(msg) && msg.split('\n').filter(l => l.trim()).length >= 3) return true; + if (/^\d+[.)]\s/m.test(msg) && msg.split('\n').filter(l => l.trim()).length >= 3) return true; + if (/^#+\s/m.test(msg)) return true; + // Uses plan mode + if (first.agentMode.includes('plan') || first.slashCommand === 'plan') return true; + return false; +} + +/* ════════════════════════════════════════════════════════════════════ + Prompt maturity helpers + ════════════════════════════════════════════════════════════════════ */ + +const CONSTRAINT_RE = /\b(must|should|shall|only|no more than|at most|at least|limit|constraint|require|restrict)\b/i; +const SUCCESS_CRITERIA_RE = /\b(expect|success|criteria|acceptance|verify|assert|should return|should output|output should|result should)\b/i; +const VERIFICATION_RE = /\b(test|verify|validate|check|confirm|ensure|assert|prove)\b/i; + +function gradePrompt(msg: string, hasFileRefs: boolean, hasCode: boolean): { + constraints: number; successCriteria: number; verificationSteps: number; + contextProvision: number; specificity: number; total: number; + issues: string[]; +} { + const issues: string[] = []; + const constraints = CONSTRAINT_RE.test(msg) ? 100 : 0; + if (!constraints) issues.push('No constraints specified'); + + const successCriteria = SUCCESS_CRITERIA_RE.test(msg) ? 100 : 0; + if (!successCriteria) issues.push('No success criteria'); + + const verificationSteps = VERIFICATION_RE.test(msg) ? 100 : 0; + if (!verificationSteps) issues.push('No verification steps'); + + let contextProvision = 0; + if (hasFileRefs) contextProvision += 50; + if (hasCode) contextProvision += 30; + if (msg.split('\n').length >= 3) contextProvision += 20; + if (contextProvision === 0) issues.push('No context provided (no file refs, no code)'); + + let specificity = 0; + if (msg.length >= 100) specificity += 40; + else if (msg.length >= 50) specificity += 20; + if (/^[-*]\s/m.test(msg) || /^\d+[.)]\s/m.test(msg)) specificity += 30; + if (msg.split('\n').filter(l => l.trim()).length >= 4) specificity += 30; + if (specificity === 0) issues.push('Very short/vague prompt'); + + const total = Math.round((constraints + successCriteria + verificationSteps + contextProvision + specificity) / 5); + return { constraints, successCriteria, verificationSteps, contextProvision, specificity, total, issues }; +} + +function scoreToGrade(score: number): 'A' | 'B' | 'C' | 'D' | 'F' { + if (score >= 80) return 'A'; + if (score >= 60) return 'B'; + if (score >= 40) return 'C'; + if (score >= 20) return 'D'; + return 'F'; +} + +/* ════════════════════════════════════════════════════════════════════ + Trend direction helper + ════════════════════════════════════════════════════════════════════ */ + +function trendDirection(values: number[]): 'stable' | 'increasing' | 'decreasing' { + if (values.length < 3) return 'stable'; + const recent = values.slice(-3); + const older = values.slice(-6, -3); + if (older.length === 0) return 'stable'; + const recentAvg = recent.reduce((a, b) => a + b, 0) / recent.length; + const olderAvg = older.reduce((a, b) => a + b, 0) / older.length; + const change = olderAvg > 0 ? (recentAvg - olderAvg) / olderAvg : 0; + if (change > 0.2) return 'increasing'; + if (change < -0.2) return 'decreasing'; + return 'stable'; +} + +/* ════════════════════════════════════════════════════════════════════ + Main Analyzer + ════════════════════════════════════════════════════════════════════ */ + +export class InsightsAnalyzer extends AnalyzerBase { + + getInsights(f?: DateFilter): InsightsData { + return { + learningVelocity: this.getLearningVelocity(f), + intentClassification: this.getIntentClassification(f), + specDriven: this.getSpecDriven(f), + productionReview: this.getProductionReview(f), + sustainablePace: this.getSustainablePace(f), + promptMaturity: this.getPromptMaturity(f), + migrationReadiness: this.getMigrationReadiness(f), + }; + } + + /* ── Learning Velocity ──────────────────────────────────────────── */ + + private getLearningVelocity(f?: DateFilter): LearningVelocityData { + const reqs = this.filter(f); + // Group languages by week + const weekLangs = new Map>(); + for (const r of reqs) { + if (!r.timestamp) continue; + const week = isoWeek(new Date(r.timestamp)); + if (!weekLangs.has(week)) weekLangs.set(week, new Set()); + const s = weekLangs.get(week)!; + for (const block of r.aiCode) if (block.language) s.add(block.language.toLowerCase()); + for (const block of r.userCode) if (block.language) s.add(block.language.toLowerCase()); + } + + const sortedWeeks = Array.from(weekLangs.keys()).sort(); + const seenBefore = new Set(); + const weeklyData: LearningVelocityData['weeklyLanguages'] = []; + const newLangCounts: number[] = []; + const cumulativeCounts: number[] = []; + + // Track first-seen per language + const langFirstSeen = new Map(); + const langWeekCount = new Map(); + + for (const week of sortedWeeks) { + const langs = Array.from(weekLangs.get(week)!); + const newLangs = langs.filter(l => !seenBefore.has(l)); + for (const l of newLangs) { + if (!langFirstSeen.has(l)) langFirstSeen.set(l, week); + } + for (const l of langs) { + langWeekCount.set(l, (langWeekCount.get(l) || 0) + 1); + seenBefore.add(l); + } + weeklyData.push({ week, languages: langs, newLanguages: newLangs }); + newLangCounts.push(newLangs.length); + cumulativeCounts.push(seenBefore.size); + } + + const topLanguages = Array.from(langFirstSeen.entries()) + .map(([language, firstSeen]) => ({ + language, + firstSeen, + weekCount: langWeekCount.get(language) || 0, + })) + .sort((a, b) => b.weekCount - a.weekCount) + .slice(0, 20); + + return { + weeklyLanguages: weeklyData, + totalLanguagesEncountered: seenBefore.size, + totalNewLanguagesLearned: newLangCounts.reduce((a, b) => a + b, 0), + velocityTrend: { + labels: sortedWeeks, + newLanguages: newLangCounts, + cumulativeLanguages: cumulativeCounts, + }, + topLanguages, + }; + } + + /* ── Intent Classification ──────────────────────────────────────── */ + + private getIntentClassification(f?: DateFilter): IntentClassificationData { + const sessions = this.filteredSessions(f); + const dist: Record = { Planning: 0, Implementation: 0, Debugging: 0, Review: 0, Exploration: 0 }; + const reqsByIntent: Record = { Planning: [], Implementation: [], Debugging: [], Review: [], Exploration: [] }; + const weeklyBuckets = new Map>(); + + const sessionIntents: IntentClassificationData['sessionIntents'] = []; + for (const s of sessions) { + const intent = classifyIntent(s); + dist[intent]++; + reqsByIntent[intent].push(s.requestCount); + + const ts = s.lastMessageDate || s.creationDate; + const date = ts != null ? toDateStr(ts) : ''; + sessionIntents.push({ + sessionId: s.sessionId, + workspaceName: s.workspaceName, + intent, + requestCount: s.requestCount, + date, + }); + + if (ts != null) { + const week = isoWeek(new Date(ts)); + if (!weeklyBuckets.has(week)) { + weeklyBuckets.set(week, { Planning: 0, Implementation: 0, Debugging: 0, Review: 0, Exploration: 0 }); + } + weeklyBuckets.get(week)![intent]++; + } + } + + const sortedWeeks = Array.from(weeklyBuckets.keys()).sort(); + const series: Record = { Planning: [], Implementation: [], Debugging: [], Review: [], Exploration: [] }; + for (const week of sortedWeeks) { + const d = weeklyBuckets.get(week)!; + for (const intent of SESSION_INTENTS) series[intent].push(d[intent]); + } + + const avgByIntent: Record = {} as Record; + for (const intent of SESSION_INTENTS) { + const arr = reqsByIntent[intent]; + avgByIntent[intent] = arr.length > 0 ? Math.round(arr.reduce((a, b) => a + b, 0) / arr.length) : 0; + } + + return { + sessionIntents, + distribution: dist, + weeklyDistribution: { labels: sortedWeeks, series }, + avgRequestsByIntent: avgByIntent, + }; + } + + /* ── Spec-Driven Flag ───────────────────────────────────────────── */ + + private getSpecDriven(f?: DateFilter): SpecDrivenData { + const sessions = this.filteredSessions(f).filter(s => s.requestCount >= 3); + const specDriven = sessions.filter(isSpecDriven); + const weekBuckets = new Map(); + + for (const s of sessions) { + const ts = s.lastMessageDate || s.creationDate; + if (ts == null) continue; + const week = isoWeek(new Date(ts)); + if (!weekBuckets.has(week)) weekBuckets.set(week, { spec: 0, unstructured: 0 }); + const b = weekBuckets.get(week)!; + if (isSpecDriven(s)) b.spec++; else b.unstructured++; + } + + const sortedWeeks = Array.from(weekBuckets.keys()).sort(); + const unstructuredExamples = sessions + .filter(s => !isSpecDriven(s)) + .slice(0, 10) + .map(s => ({ + workspaceName: s.workspaceName, + firstPrompt: s.requests[0]?.messageText.substring(0, 120) || '', + date: s.lastMessageDate != null ? toDateStr(s.lastMessageDate) : '', + })); + + return { + totalSessions: sessions.length, + specDrivenCount: specDriven.length, + specDrivenRate: sessions.length > 0 ? specDriven.length / sessions.length : 0, + weeklyTrend: { + labels: sortedWeeks, + specDriven: sortedWeeks.map(w => weekBuckets.get(w)!.spec), + unstructured: sortedWeeks.map(w => weekBuckets.get(w)!.unstructured), + }, + unstructuredExamples, + }; + } + + /* ── Production vs Review Ratio ─────────────────────────────────── */ + + private getProductionReview(f?: DateFilter): ProductionReviewData { + const sessions = this.filteredSessions(f); + let totalAiLoc = 0; + let reviewedLoc = 0; + const reviewGaps: number[] = []; + let sessionsNoReview = 0; + + const weekBuckets = new Map(); + + for (const s of sessions) { + let sessionReviewedLoc = 0; + let sessionAiLoc = 0; + const ts = s.lastMessageDate || s.creationDate; + const week = ts != null ? isoWeek(new Date(ts)) : null; + if (week && !weekBuckets.has(week)) weekBuckets.set(week, { produced: 0, reviewed: 0 }); + + for (let i = 0; i < s.requests.length; i++) { + const r = s.requests[i]; + const aiLoc = r.aiCode.reduce((sum, b) => sum + b.loc, 0); + sessionAiLoc += aiLoc; + + const gap = aiLoc > 0 && i < s.requests.length - 1 + ? this.getReviewGap(r, s.requests[i + 1]) + : null; + if (gap != null && gap >= REVIEW_GAP_THRESHOLD_MS) { + sessionReviewedLoc += aiLoc; + reviewGaps.push(gap); + } + } + + totalAiLoc += sessionAiLoc; + reviewedLoc += sessionReviewedLoc; + if (sessionAiLoc > 0 && sessionReviewedLoc === 0) sessionsNoReview++; + + if (week) { + const b = weekBuckets.get(week)!; + b.produced += sessionAiLoc; + b.reviewed += sessionReviewedLoc; + } + } + + const sortedWeeks = Array.from(weekBuckets.keys()).sort(); + const avgGap = reviewGaps.length > 0 + ? Math.round(reviewGaps.reduce((a, b) => a + b, 0) / reviewGaps.length / 1000) + : 0; + + return { + totalAiLoc, + estimatedReviewedLoc: reviewedLoc, + reviewRatio: totalAiLoc > 0 ? reviewedLoc / totalAiLoc : 0, + weeklyTrend: { + labels: sortedWeeks, + produced: sortedWeeks.map(w => weekBuckets.get(w)!.produced), + estimated_reviewed: sortedWeeks.map(w => weekBuckets.get(w)!.reviewed), + }, + sessionsWithoutReview: sessionsNoReview, + avgReviewGapSec: avgGap, + }; + } + + /* ── Sustainable Pace ───────────────────────────────────────────── */ + + private getSustainablePace(f?: DateFilter): SustainablePaceData { + const reqs = this.filter(f); + const sessions = this.filteredSessions(f); + const weekBuckets = this.buildSustainablePaceBuckets(reqs, sessions); + const sortedWeeks = Array.from(weekBuckets.keys()).sort(); + const lateNightReqs = sortedWeeks.map(w => weekBuckets.get(w)!.lateNight); + const weekendReqs = sortedWeeks.map(w => weekBuckets.get(w)!.weekend); + const totalReqs = sortedWeeks.map(w => weekBuckets.get(w)!.total); + const avgSessionLength = sortedWeeks.map(w => { + const lengths = weekBuckets.get(w)!.sessionLengths; + return lengths.length > 0 ? Math.round(lengths.reduce((a, b) => a + b, 0) / lengths.length) : 0; + }); + const currentStreak = this.getCurrentActiveDayStreak(reqs); + const lateNightTrending = trendDirection(lateNightReqs); + const weekendTrending = trendDirection(weekendReqs); + const alerts = this.getSustainablePaceAlerts(reqs, currentStreak, lateNightReqs, weekendReqs, lateNightTrending, weekendTrending); + + return { + weeklyTrend: { labels: sortedWeeks, lateNightReqs, weekendReqs, totalReqs, avgSessionLength }, + burnoutRisk: this.getBurnoutRisk(alerts, currentStreak, lateNightTrending), + alerts, + currentStreak, + weekendTrending, + lateNightTrending, + }; + } + + /* ── Prompt Engineering Maturity ────────────────────────────────── */ + + private getPromptMaturity(f?: DateFilter): PromptMaturityData { + const reqs = this.filter(f).filter(r => r.messageLength > 10); + if (reqs.length === 0) { + return { + overallGrade: 'F', score: 0, + dimensions: { constraints: 0, successCriteria: 0, verificationSteps: 0, contextProvision: 0, specificity: 0 }, + weeklyTrend: { labels: [], scores: [] }, + samplePrompts: [], + }; + } + + // Sample prompts evenly + const step = Math.max(1, Math.floor(reqs.length / PROMPT_MATURITY_SAMPLE_SIZE)); + const sampled = reqs.filter((_, i) => i % step === 0).slice(0, PROMPT_MATURITY_SAMPLE_SIZE); + + const totals = { constraints: 0, successCriteria: 0, verificationSteps: 0, contextProvision: 0, specificity: 0 }; + + const graded = sampled.map(r => { + const hasFileRefs = r.referencedFiles.length > 0 || (r.variableKinds['file'] > 0); + const hasCode = r.userCode.length > 0; + const g = gradePrompt(r.messageText, hasFileRefs, hasCode); + totals.constraints += g.constraints; + totals.successCriteria += g.successCriteria; + totals.verificationSteps += g.verificationSteps; + totals.contextProvision += g.contextProvision; + totals.specificity += g.specificity; + return { text: r.messageText.substring(0, 150), grade: scoreToGrade(g.total), issues: g.issues, total: g.total, timestamp: r.timestamp }; + }); + + const n = sampled.length; + const dims = { + constraints: Math.round(totals.constraints / n), + successCriteria: Math.round(totals.successCriteria / n), + verificationSteps: Math.round(totals.verificationSteps / n), + contextProvision: Math.round(totals.contextProvision / n), + specificity: Math.round(totals.specificity / n), + }; + const overall = Math.round((dims.constraints + dims.successCriteria + dims.verificationSteps + dims.contextProvision + dims.specificity) / 5); + + // Weekly trend + const weekScores = new Map(); + for (const r of reqs) { + if (!r.timestamp) continue; + const week = isoWeek(new Date(r.timestamp)); + if (!weekScores.has(week)) weekScores.set(week, []); + const hasFileRefs = r.referencedFiles.length > 0 || (r.variableKinds['file'] > 0); + const hasCode = r.userCode.length > 0; + weekScores.get(week)!.push(gradePrompt(r.messageText, hasFileRefs, hasCode).total); + } + const sortedWeeks = Array.from(weekScores.keys()).sort(); + const weeklyScores = sortedWeeks.map(w => { + const arr = weekScores.get(w)!; + return Math.round(arr.reduce((a, b) => a + b, 0) / arr.length); + }); + + // Pick worst prompts as examples + const samplePrompts = graded + .sort((a, b) => a.total - b.total) + .slice(0, 5) + .map(g => ({ text: g.text, grade: g.grade, issues: g.issues })); + + return { + overallGrade: scoreToGrade(overall), + score: overall, + dimensions: dims, + weeklyTrend: { labels: sortedWeeks, scores: weeklyScores }, + samplePrompts, + }; + } + + /* ── Migration Readiness ────────────────────────────────────────── */ + + private getMigrationReadiness(f?: DateFilter): MigrationReadinessData { + const sessions = this.filteredSessions(f); + if (sessions.length === 0) { + return { primaryHarness: '', missingFeatures: [], readinessScore: 0, featureUsage: [] }; + } + + const harnessCounts = new Map(); + for (const session of sessions) harnessCounts.set(session.harness, (harnessCounts.get(session.harness) || 0) + 1); + const sortedHarnesses = Array.from(harnessCounts.entries()).sort((a, b) => b[1] - a[1]); + const primaryHarness = sortedHarnesses[0][0]; + const allHarnesses = sortedHarnesses.map(([harness]) => harness); + const featureMatrix = this.getMigrationFeatureMatrix(); + const usedFeatures = this.getUsedMigrationFeatures(this.filter(f), sessions); + const missingFeatures = featureMatrix + .filter(feature => !usedFeatures.has(feature.feature)) + .map(feature => ({ + feature: feature.feature, + availableIn: feature.harnesses.filter(harness => harness !== primaryHarness && allHarnesses.includes(harness)), + description: feature.description, + })) + .filter(feature => feature.availableIn.length > 0); + const featureUsage = featureMatrix + .filter(feature => feature.harnesses.some(harness => allHarnesses.includes(harness))) + .map(feature => ({ + feature: feature.feature, + used: usedFeatures.has(feature.feature), + harnesses: feature.harnesses.filter(harness => allHarnesses.includes(harness)), + })); + const readinessScore = featureMatrix.length > 0 ? Math.round((usedFeatures.size / featureMatrix.length) * 100) : 0; + + return { primaryHarness, missingFeatures, readinessScore, featureUsage }; + } + + private getReviewGap(current: Session['requests'][number], next: Session['requests'][number]): number | null { + if (!current.timestamp || !next.timestamp || !current.totalElapsed) return null; + return next.timestamp - (current.timestamp + current.totalElapsed); + } + + private buildSustainablePaceBuckets( + reqs: Session['requests'], + sessions: Session[], + ): Map { + const weekBuckets = new Map(); + for (const request of reqs) { + if (!request.timestamp) continue; + const date = new Date(request.timestamp); + const week = isoWeek(date); + if (!weekBuckets.has(week)) weekBuckets.set(week, { total: 0, lateNight: 0, weekend: 0, sessionLengths: [] }); + const bucket = weekBuckets.get(week)!; + bucket.total++; + const hour = date.getHours(); + if (hour >= LATE_NIGHT_START || hour < LATE_NIGHT_END) bucket.lateNight++; + const dayOfWeek = date.getDay(); + if (dayOfWeek === 0 || dayOfWeek === 6) bucket.weekend++; + } + for (const session of sessions) { + const ts = session.lastMessageDate || session.creationDate; + if (ts == null) continue; + const week = isoWeek(new Date(ts)); + if (!weekBuckets.has(week)) weekBuckets.set(week, { total: 0, lateNight: 0, weekend: 0, sessionLengths: [] }); + weekBuckets.get(week)!.sessionLengths.push(session.requestCount); + } + return weekBuckets; + } + + private getCurrentActiveDayStreak(reqs: Session['requests']): number { + const activeDays = new Set(); + for (const request of reqs) { + if (request.timestamp) activeDays.add(toDateStr(request.timestamp)); + } + const sortedDays = Array.from(activeDays).sort().reverse(); + let currentStreak = 0; + let checkDate = toDateStr(Date.now()); + for (const day of sortedDays) { + if (day !== checkDate) break; + currentStreak++; + const date = new Date(checkDate + 'T00:00:00'); + date.setDate(date.getDate() - 1); + checkDate = toDateStr(date.getTime()); + } + return currentStreak; + } + + private getSustainablePaceAlerts( + reqs: Session['requests'], + currentStreak: number, + lateNightReqs: number[], + weekendReqs: number[], + lateNightTrending: 'stable' | 'increasing' | 'decreasing', + weekendTrending: 'stable' | 'increasing' | 'decreasing', + ): string[] { + const alerts: string[] = []; + if (currentStreak >= BURNOUT_STREAK_DAYS) { + alerts.push(`Working ${currentStreak} consecutive days without a break. Consider taking a day off.`); + } + if (lateNightTrending === 'increasing') { + alerts.push('Late-night sessions are increasing week over week. This is a burnout risk factor.'); + } + if (weekendTrending === 'increasing') { + alerts.push('Weekend work is creeping up. Monitor your work-life boundaries.'); + } + + const totalReqs = reqs.length; + const lateNightRate = totalReqs > 0 ? lateNightReqs.reduce((a, b) => a + b, 0) / totalReqs : 0; + const weekendRate = totalReqs > 0 ? weekendReqs.reduce((a, b) => a + b, 0) / totalReqs : 0; + if (lateNightRate > BURNOUT_LATE_NIGHT_RATE) { + alerts.push(`${(lateNightRate * 100).toFixed(0)}% of requests happen during late-night hours (${LATE_NIGHT_START}:00-${LATE_NIGHT_END}:00).`); + } + if (weekendRate > BURNOUT_WEEKEND_RATE) { + alerts.push(`${(weekendRate * 100).toFixed(0)}% of requests happen on weekends.`); + } + return alerts; + } + + private getBurnoutRisk( + alerts: string[], + currentStreak: number, + lateNightTrending: 'stable' | 'increasing' | 'decreasing', + ): 'low' | 'medium' | 'high' { + if (alerts.length >= 3 || (currentStreak >= BURNOUT_STREAK_DAYS && lateNightTrending === 'increasing')) { + return 'high'; + } + if (alerts.length >= 1) return 'medium'; + return 'low'; + } + + private getMigrationFeatureMatrix(): Array<{ feature: string; description: string; harnesses: string[] }> { + return [ + { feature: 'Sub-agents', description: 'Delegate sub-tasks to specialized agents', harnesses: ['Local Agent', 'Local Agent (Insiders)', 'Claude'] }, + { feature: 'MCP Tools', description: 'Model Context Protocol tool integration', harnesses: ['Local Agent', 'Local Agent (Insiders)', 'Claude'] }, + { feature: 'Custom Instructions', description: 'Project-level AI instructions (.instructions.md)', harnesses: ['Local Agent', 'Local Agent (Insiders)', 'Claude'] }, + { feature: 'Plan Mode', description: 'Separate planning step before implementation', harnesses: ['Local Agent', 'Local Agent (Insiders)', 'Claude'] }, + { feature: 'Skills', description: 'Domain-specific knowledge modules', harnesses: ['Local Agent', 'Local Agent (Insiders)'] }, + { feature: 'Slash Commands', description: '/fix, /explain, /tests, /doc', harnesses: ['Local Agent', 'Local Agent (Insiders)'] }, + { feature: 'Multi-file Edits', description: 'Edit multiple files in a single turn', harnesses: ['Local Agent', 'Local Agent (Insiders)', 'Claude', 'Codex'] }, + { feature: 'Terminal Access', description: 'Run commands as part of agent workflow', harnesses: ['Local Agent', 'Local Agent (Insiders)', 'Claude', 'Codex', 'OpenCode'] }, + { feature: 'File References', description: 'Reference specific files in prompts', harnesses: ['Local Agent', 'Local Agent (Insiders)', 'Claude'] }, + { feature: 'Parallel Sessions', description: 'Run multiple conversations simultaneously', harnesses: ['Local Agent', 'Local Agent (Insiders)', 'Claude', 'Codex'] }, + ]; + } + + private getUsedMigrationFeatures(reqs: Session['requests'], sessions: Session[]): Set { + const usedFeatures = new Set(); + for (const request of reqs) { + if (request.agentName && request.agentName !== 'copilot') usedFeatures.add('Sub-agents'); + if (request.toolsUsed.some(tool => tool.startsWith('mcp_'))) usedFeatures.add('MCP Tools'); + if (request.customInstructions.length > 0) usedFeatures.add('Custom Instructions'); + if (request.agentMode.includes('plan') || request.slashCommand === 'plan') usedFeatures.add('Plan Mode'); + if (request.skillsUsed.length > 0) usedFeatures.add('Skills'); + if (request.slashCommand) usedFeatures.add('Slash Commands'); + if (request.editedFiles.length > 1) usedFeatures.add('Multi-file Edits'); + if (request.toolsUsed.some(tool => tool.includes('terminal') || tool.includes('runCommand'))) usedFeatures.add('Terminal Access'); + if (request.referencedFiles.length > 0 || (request.variableKinds['file'] > 0)) usedFeatures.add('File References'); + } + const daySessions = new Map(); + for (const session of sessions) { + const ts = session.lastMessageDate || session.creationDate; + if (ts == null) continue; + const day = toDateStr(ts); + daySessions.set(day, (daySessions.get(day) || 0) + 1); + } + if (Array.from(daySessions.values()).some(count => count >= 2)) usedFeatures.add('Parallel Sessions'); + return usedFeatures; + } +} diff --git a/packages/ai-coach-analytics/src/cache.ts b/packages/ai-coach-analytics/src/cache.ts new file mode 100644 index 0000000..191f3bf --- /dev/null +++ b/packages/ai-coach-analytics/src/cache.ts @@ -0,0 +1,12 @@ +/** + * Minimal stand-in for the original cache.ts. + * Only the SessionSource type is needed by parser-shared.ts in this package; + * the disk-cache implementation (which depends on VS Code-specific parsers) is not ported. + */ +export interface SessionSource { + kind: "vscode-session-file" | "cli-events"; + filePath: string; + workspaceId: string; + workspaceName: string; + harness: string; +} diff --git a/packages/ai-coach-analytics/src/constants.ts b/packages/ai-coach-analytics/src/constants.ts new file mode 100644 index 0000000..e0e8457 --- /dev/null +++ b/packages/ai-coach-analytics/src/constants.ts @@ -0,0 +1,127 @@ +/*--------------------------------------------------------------------------------------------- + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the MIT License. See LICENSE in the project root for license information. + *--------------------------------------------------------------------------------------------*/ + +/* Shared constants for AI Engineer Coach */ + +/* ---- Model multipliers for premium request cost ---- */ +/* https://docs.github.com/en/copilot/concepts/billing/copilot-requests#model-multipliers */ +export const MODEL_MULTIPLIERS: Record = { + 'gpt-4.1': 0, 'gpt-4.1-mini': 0, 'gpt-4.1-nano': 0, + 'gpt-5-mini': 0, 'gpt-5.1': 1, 'gpt-5.1-codex': 1, 'gpt-5.1-codex-mini': 0.33, + 'gpt-5.1-codex-max': 1, 'gpt-5.2': 1, 'gpt-5.2-codex': 1, + 'gpt-5.3-codex': 1, 'gpt-5.4': 1, 'gpt-5.4-mini': 0.33, + 'o1': 1, 'o1-mini': 1, 'o1-preview': 1, 'o3': 1, 'o3-mini': 1, 'o4-mini': 1, + 'claude-3.5-sonnet': 1, 'claude-3.7-sonnet': 1, 'claude-sonnet-4': 1, 'claude-sonnet-4.5': 1, + 'claude-sonnet-4.6': 1, 'claude-4': 1, 'claude-opus-4': 1, 'claude-opus-41': 1, + 'claude-opus-4.5': 3, 'claude-opus-4.6': 3, 'claude-opus-4.6-fast': 30, 'claude-opus-4.6-1m': 3, + 'claude-opus-4.7': 7.5, + 'claude-haiku-4.5': 0.33, + 'gemini-2.0-flash': 0.25, 'gemini-2.5-pro': 1, 'gemini-3-flash': 0.33, + 'gemini-3-pro': 1, 'gemini-3.1-pro': 1, + 'grok-code-fast-1': 0.25, 'raptor-mini': 0, 'goldeneye': 1, + 'copilot-internal': 0, 'auto': 1, 'custom-model': 1, +}; + +export const LOC_COST_2010 = 20; + +/* ---- Per-token rates in USD per 1M tokens (May 2026 pricing) ---- */ +/* https://docs.github.com/en/copilot/reference/copilot-billing/models-and-pricing */ +export interface TokenRate { input: number; cached: number; output: number; cacheWrite?: number } +export const MODEL_TOKEN_RATES: Record = { + 'gpt-4.1': { input: 2.00, cached: 0.50, output: 8.00 }, + 'gpt-4.1-mini': { input: 0.25, cached: 0.025, output: 2.00 }, + 'gpt-4.1-nano': { input: 0.20, cached: 0.02, output: 1.25 }, + 'gpt-5-mini': { input: 0.25, cached: 0.025, output: 2.00 }, + 'gpt-5.1': { input: 1.75, cached: 0.175, output: 14.00 }, + 'gpt-5.1-codex': { input: 1.75, cached: 0.175, output: 14.00 }, + 'gpt-5.1-codex-mini': { input: 0.25, cached: 0.025, output: 2.00 }, + 'gpt-5.1-codex-max': { input: 1.75, cached: 0.175, output: 14.00 }, + 'gpt-5.2': { input: 1.75, cached: 0.175, output: 14.00 }, + 'gpt-5.2-codex': { input: 1.75, cached: 0.175, output: 14.00 }, + 'gpt-5.3-codex': { input: 1.75, cached: 0.175, output: 14.00 }, + 'gpt-5.4': { input: 2.50, cached: 0.25, output: 15.00 }, + 'gpt-5.4-mini': { input: 0.75, cached: 0.075, output: 4.50 }, + 'gpt-5.4-nano': { input: 0.20, cached: 0.02, output: 1.25 }, + 'gpt-5.5': { input: 5.00, cached: 0.50, output: 30.00 }, + 'claude-haiku-4.5': { input: 1.00, cached: 0.10, output: 5.00, cacheWrite: 1.25 }, + 'claude-3.5-sonnet': { input: 3.00, cached: 0.30, output: 15.00, cacheWrite: 3.75 }, + 'claude-3.7-sonnet': { input: 3.00, cached: 0.30, output: 15.00, cacheWrite: 3.75 }, + 'claude-sonnet-4': { input: 3.00, cached: 0.30, output: 15.00, cacheWrite: 3.75 }, + 'claude-sonnet-4.5': { input: 3.00, cached: 0.30, output: 15.00, cacheWrite: 3.75 }, + 'claude-sonnet-4.6': { input: 3.00, cached: 0.30, output: 15.00, cacheWrite: 3.75 }, + 'claude-opus-4.5': { input: 5.00, cached: 0.50, output: 25.00, cacheWrite: 6.25 }, + 'claude-opus-4.6': { input: 5.00, cached: 0.50, output: 25.00, cacheWrite: 6.25 }, + 'claude-opus-4.7': { input: 5.00, cached: 0.50, output: 25.00, cacheWrite: 6.25 }, + 'gemini-2.0-flash': { input: 0.50, cached: 0.05, output: 3.00 }, + 'gemini-2.5-pro': { input: 1.25, cached: 0.125, output: 10.00 }, + 'gemini-3-flash': { input: 0.50, cached: 0.05, output: 3.00 }, + 'gemini-3-pro': { input: 2.00, cached: 0.20, output: 12.00 }, + 'gemini-3.1-pro': { input: 2.00, cached: 0.20, output: 12.00 }, + 'grok-code-fast-1': { input: 0.20, cached: 0.02, output: 1.50 }, + 'raptor-mini': { input: 0.25, cached: 0.025, output: 2.00 }, + 'goldeneye': { input: 1.25, cached: 0.125, output: 10.00 }, +}; + +/* AI Credit budget per SKU (1 credit = $0.01 USD) */ +export const SKU_AI_CREDITS: Record = { + 'pro': 1000, + 'pro-plus': 3900, + 'business': 1900, + 'enterprise': 3900, +}; + +/* ---- Legacy threshold kept for analyzer-patterns.ts ---- */ +export const LONG_SESSION_REQS = 30; + +/* ---- Insights thresholds ---- */ +export const REVIEW_GAP_THRESHOLD_MS = 30_000; // 30s gap after AI code = likely reviewed +export const VIBE_CODE_MIN_LOC = 100; // min AI LoC per session to flag +export const VIBE_CODE_MAX_USER_PROMPTS = 5; // max user prompts in a vibe-coded session +export const VIBE_CODE_MIN_SESSIONS = 3; // min sessions to flag +export const CONTEXT_AUDIT_MIN_REQS = 30; // min requests for context audit +export const PROMPT_MATURITY_SAMPLE_SIZE = 50; // prompts to sample for maturity grading +export const LATE_NIGHT_START = 22; // 10 PM +export const LATE_NIGHT_END = 5; // 5 AM +export const BURNOUT_STREAK_DAYS = 14; // consecutive days threshold +export const BURNOUT_LATE_NIGHT_RATE = 0.15; // late-night ratio threshold +export const BURNOUT_WEEKEND_RATE = 0.25; // weekend ratio threshold + +/* ---- Config health thresholds ---- */ +export const LOW_CONSTRAINT_MIN_REQS = 30; // min requests to flag low constraint usage +export const LOW_CONSTRAINT_RATE = 0.08; // <8% of prompts use constraints → anti-pattern +export const LOW_MARKDOWN_RATIO = 0.05; // <5% markdown LoC → likely no spec-driven development +export const LOW_MARKDOWN_MIN_LOC = 100; // min total AI LoC in a workspace to flag +export const LOW_MARKDOWN_MIN_WORKSPACES = 1; // min workspaces with low markdown to flag +export const OVERSIZED_INSTRUCTION_LINES = 500; // instruction file too large (lines) +export const COPILOT_INSTRUCTION_MAX_CHARS = 4000; // Copilot code review truncation limit +export const CLAUDE_MD_RECOMMENDED_LINES = 200; // CLAUDE.md recommended max lines + +/* ---- Flow state thresholds ---- */ +export const FLOW_RAPID_FOLLOWUP_SEC = 30; // follow-up within 30s = rapid (in the zone) +export const FLOW_SESSION_MIN_REQS = 3; // min requests for a session to count +export const FLOW_BLOCK_GAP_MIN = 15; // gap > 15 min between requests = new work block +export const FLOW_DEEP_SCORE = 70; // 70-100 = deep flow +export const FLOW_MODERATE_SCORE = 45; // 45-69 = moderate flow +export const FLOW_SHALLOW_SCORE = 25; // 25-44 = shallow | <25 = fragmented +export const FLOW_LOW_SCORE_RATE = 0.6; // >60% fragmented days → anti-pattern +export const FLOW_MIN_DAYS = 5; // min days of activity to flag + +/* ---- Context management thresholds ---- */ +export const CONTEXT_WINDOW_DEFAULT = 128_000; // default assumed context window (tokens) +export const CONTEXT_OPTIMAL_UTILIZATION = 50; // ≤50% avg = optimal (top performance) +export const CONTEXT_LIMITED_UTILIZATION = 80; // >80% avg = limited +export const CONTEXT_SATURATION_THRESHOLD = 60; // requests ≥60% utilization count as saturated +export const CONTEXT_COMPACTION_STORM_MIN = 4; // 4+ compactions in a session = storm +export const CONTEXT_MIN_TOKEN_REQUESTS = 5; // min requests with tokens to score a session +export const CONTEXT_GROWING_SESSION_MIN_REQS = 8; // min requests to detect runaway growth +export const CONTEXT_GROWING_SESSION_GROWTH_RATE = 0.8; // 80%+ sequential increases = runaway + +/* ---- Token estimation (for sessions without native token data) ---- */ + +/* ---- Token data quality cutoff ---- */ +export const TOKEN_DATA_AVAILABLE_FROM = '2026-04-01'; + +/* ---- Feature flags ---- */ +export const FF_TOKEN_REPORTING_ENABLED = false; diff --git a/packages/ai-coach-analytics/src/detector-registry.ts b/packages/ai-coach-analytics/src/detector-registry.ts new file mode 100644 index 0000000..e783f43 --- /dev/null +++ b/packages/ai-coach-analytics/src/detector-registry.ts @@ -0,0 +1,306 @@ +/*--------------------------------------------------------------------------------------------- + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the MIT License. See LICENSE in the project root for license information. + *--------------------------------------------------------------------------------------------*/ + +import { AntiPattern, OccurrenceDetail, PracticeGroup, Session, SessionRequest } from './types'; +import type { DetectionRule, DetectorEmission, RuleTemplateVars } from './types'; +import { getAllRules } from './rule-engine'; +import { registerAllBuiltinRules, loadPersonalRules, registerAllBuiltinMetrics } from './rule-loader'; +import { fillTemplate } from './rule-parser'; +import { parsePipeline, executePipeline, checkPipelineTrigger, resolveInheritance } from './rule-pipeline'; +import { isoWeek } from './helpers'; + +registerAllBuiltinRules(); +registerAllBuiltinMetrics(); +loadPersonalRules(); + +interface DetectorContext { + reqs: SessionRequest[]; + sessions: Session[]; + skipIdeDetectors: boolean; +} + +export interface DetectorDefinition { + name: string; + group: PracticeGroup; + requiresIdeContext?: boolean; + rule?: DetectionRule; + run: (ctx: DetectorContext) => AntiPattern | null; +} + +/* ------------------------------------------------------------------ */ +/* Map emission + rule → AntiPattern (the output format callers need) */ +/* ------------------------------------------------------------------ */ + +const MAX_DETAILS = 200; + +type DetailRow = Record; + +function textValue(value: unknown): string { + if (typeof value === 'string') return value; + if (typeof value === 'number' || typeof value === 'boolean' || typeof value === 'bigint') { + return `${value}`; + } + return ''; +} + +function truncatedText(value: unknown): string { + return textValue(value).substring(0, 120); +} + +function numericValue(value: unknown): number { + return typeof value === 'number' ? value : Number(value ?? 0); +} + +function recordValue(value: unknown): DetailRow | null { + return typeof value === 'object' && value !== null ? value as DetailRow : null; +} + +function addWeekCount(weekCounts: Map, timestamp: number): void { + if (timestamp <= 0) return; + const wk = isoWeek(new Date(timestamp)); + weekCounts.set(wk, (weekCounts.get(wk) || 0) + 1); +} + +function collectWorkspaceStats(row: DetailRow): Record { + const stats: Record = {}; + for (const [key, value] of Object.entries(row)) { + if (key !== 'name' && key !== 'workspace' && key !== 'isLow' && typeof value === 'number') { + stats[key] = value; + } + } + return stats; +} + +function addWorkspaceDetails(extraWorkspaces: unknown, details: OccurrenceDetail[]): boolean { + if (!Array.isArray(extraWorkspaces) || extraWorkspaces.length === 0) return false; + for (const item of extraWorkspaces) { + if (details.length >= MAX_DETAILS) break; + const row = recordValue(item); + if (!row) continue; + details.push({ + timestamp: 0, + workspace: textValue(row.name) || textValue(row.workspace), + sessionId: '', + message: '', + model: '', + kind: 'workspace', + stats: collectWorkspaceStats(row), + }); + } + return true; +} + +function firstRequestMessage(row: DetailRow): string { + if (!Array.isArray(row.requests) || row.requests.length === 0) return ''; + const first = recordValue(row.requests[0]); + return first ? truncatedText(first.messageText) : ''; +} + +function addSessionDetail(row: DetailRow, details: OccurrenceDetail[], weekCounts: Map): void { + const timestamp = numericValue(row.creationDate ?? row.lastMessageDate ?? row.startTime); + addWeekCount(weekCounts, timestamp); + if (details.length >= MAX_DETAILS) return; + details.push({ + timestamp, + workspace: textValue(row.workspaceName) || textValue(row.workspaceId), + sessionId: textValue(row.sessionId), + message: firstRequestMessage(row), + model: '', + }); +} + +function addRequestDetail(row: DetailRow, details: OccurrenceDetail[], weekCounts: Map): void { + const timestamp = numericValue(row.timestamp); + addWeekCount(weekCounts, timestamp); + if (details.length >= MAX_DETAILS) return; + details.push({ + timestamp, + workspace: textValue(row.workspaceName) || textValue(row.sessionId), + sessionId: textValue(row.sessionId), + message: truncatedText(row.messageText), + model: textValue(row.modelId), + }); +} + +function collectMatchedRowDetails( + emission: DetectorEmission, + rule: DetectionRule, + details: OccurrenceDetail[], + weekCounts: Map, +): void { + if (!emission.matchedRows) return; + const addDetail = rule.scope === 'sessions' ? addSessionDetail : addRequestDetail; + for (const row of emission.matchedRows) { + addDetail(row, details, weekCounts); + } +} + +function emissionToAntiPattern( + emission: DetectorEmission, + rule: DetectionRule, +): AntiPattern { + const pct = emission.total > 0 + ? `${(emission.count / emission.total * 100).toFixed(0)}%` + : '0%'; + + const vars: RuleTemplateVars = { + count: emission.count, + total: emission.total, + ratio: emission.ratio, + pct, + extra: { ...rule.thresholds, ...emission.extra }, + }; + + const severity = emission.dynamicSeverity ?? rule.severity; + const description = rule.descriptionTemplate + ? fillTemplate(rule.descriptionTemplate, vars as unknown as Record) + : `${rule.name}: ${emission.count} occurrences`; + const suggestion = rule.suggestionTemplate + ? fillTemplate(rule.suggestionTemplate, vars as unknown as Record) + : ''; + + const details: OccurrenceDetail[] = []; + const weekCounts = new Map(); + const hasWorkspaceDetails = addWorkspaceDetails(emission.extra?.workspaces, details); + if (!hasWorkspaceDetails) { + collectMatchedRowDetails(emission, rule, details, weekCounts); + } + + details.sort((a, b) => b.timestamp - a.timestamp); + + return { + id: rule.id, + name: rule.name, + severity, + group: rule.group, + occurrences: emission.count, + description, + suggestion, + examples: emission.examples, + details, + weeklyHist: buildWeeklyHist(weekCounts), + }; +} + +/** Build a weekly histogram with continuous labels (fill gaps with 0). */ +function buildWeeklyHist(weekCounts: Map): { labels: string[]; counts: number[] } { + if (weekCounts.size === 0) return { labels: [], counts: [] }; + const sorted = [...weekCounts.keys()].sort(); + const first = sorted[0]; + const last = sorted[sorted.length - 1]; + + // Generate continuous week labels between first and last + const labels: string[] = []; + const counts: number[] = []; + const d = parseIsoWeek(first); + const end = parseIsoWeek(last); + while (d <= end) { + const wk = isoWeek(d); + labels.push(wk); + counts.push(weekCounts.get(wk) || 0); + d.setDate(d.getDate() + 7); + } + // Limit to last 12 weeks + const MAX_WEEKS = 12; + if (labels.length > MAX_WEEKS) { + const start = labels.length - MAX_WEEKS; + return { labels: labels.slice(start), counts: counts.slice(start) }; + } + return { labels, counts }; +} + +/** Parse an ISO week string like "2025-W14" to a Date (Monday of that week). */ +function parseIsoWeek(wk: string): Date { + const [yearStr, weekStr] = wk.split('-W'); + const year = Number.parseInt(yearStr, 10); + const week = Number.parseInt(weekStr, 10); + const jan1 = new Date(year, 0, 1); + const dayOfWeek = jan1.getDay() || 7; // ISO: Monday = 1 + const firstMonday = new Date(year, 0, 1 + (8 - dayOfWeek) % 7); + const d = new Date(firstMonday); + d.setDate(d.getDate() + (week - 1) * 7); + return d; +} + +/* ------------------------------------------------------------------ */ +/* Pipeline-based registry build */ +/* ------------------------------------------------------------------ */ + +function buildRegistry(): DetectorDefinition[] { + const rules = getAllRules(); + const registry: DetectorDefinition[] = []; + + for (const rawRule of rules) { + const rule = resolveInheritance(rawRule); + const pipeline = parsePipeline(rule); + + registry.push({ + name: rule.name, + group: rule.group, + requiresIdeContext: rule.requiresIdeContext, + rule, + run: (ctx) => { + const emission = executePipeline(pipeline, rule, ctx); + if (!checkPipelineTrigger(pipeline, emission, rule)) return null; + return emissionToAntiPattern(emission, rule); + }, + }); + } + + return registry; +} + +let _registry: DetectorDefinition[] | null = null; + +export const DETECTOR_REGISTRY: DetectorDefinition[] = new Proxy([] as DetectorDefinition[], { + get(_target, prop) { + if (!_registry) _registry = buildRegistry(); + return Reflect.get(_registry, prop) as unknown; + }, +}); + +export function invalidateDetectorRegistry(): void { + _registry = null; +} + +export function getActiveDetectors(skipIdeDetectors: boolean): DetectorDefinition[] { + const registry = buildRegistry(); + return registry.filter(detector => !skipIdeDetectors || !detector.requiresIdeContext); +} + +export function runDetectors(reqs: SessionRequest[], sessions: Session[], skipIdeDetectors: boolean): AntiPattern[] { + return getActiveDetectors(skipIdeDetectors) + .map(detector => detector.run({ reqs, sessions, skipIdeDetectors })) + .filter((pattern): pattern is AntiPattern => pattern !== null); +} + +/** + * Run emitters only (no trigger evaluation). Used by rule-engine preview. + */ +export function runEmitters(reqs: SessionRequest[], sessions: Session[], skipIdeDetectors: boolean): Map { + const rules = getAllRules(); + const results = new Map(); + for (const rawRule of rules) { + if (skipIdeDetectors && rawRule.requiresIdeContext) continue; + const rule = resolveInheritance(rawRule); + const pipeline = parsePipeline(rule); + results.set(rule.id, executePipeline(pipeline, rule, { reqs, sessions, skipIdeDetectors })); + } + return results; +} + +export function getDetectorGroupCounts(skipIdeDetectors: boolean): Record { + const counts: Record = { + 'prompt-quality': 0, + 'session-hygiene': 0, + 'code-review': 0, + 'tool-mastery': 0, + 'context-management': 0, + }; + for (const detector of getActiveDetectors(skipIdeDetectors)) { + counts[detector.group] += 1; + } + return counts; +} \ No newline at end of file diff --git a/packages/ai-coach-analytics/src/detectors/scoring.ts b/packages/ai-coach-analytics/src/detectors/scoring.ts new file mode 100644 index 0000000..28620f8 --- /dev/null +++ b/packages/ai-coach-analytics/src/detectors/scoring.ts @@ -0,0 +1,94 @@ +/*--------------------------------------------------------------------------------------------- + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the MIT License. See LICENSE in the project root for license information. + *--------------------------------------------------------------------------------------------*/ + +import { SessionRequest, PracticeGroup } from '../types'; +import { isoWeek } from '../helpers'; + +function promptQualityPenalty(request: SessionRequest): number { + let penalty = 0; + if (request.messageLength < 30 && request.messageLength > 0) penalty += 1; + if (request.referencedFiles.length === 0 && !(request.variableKinds['file'] > 0) && request.editedFiles.length === 0) penalty += 0.5; + return penalty; +} + +function sessionHygienePenalty(request: SessionRequest): number { + let penalty = 0; + if (request.isCanceled) penalty += 1; + if (request.timestamp !== null) { + const when = new Date(request.timestamp); + if (when.getHours() < 5) penalty += 0.3; + const dayOfWeek = when.getDay(); + if (dayOfWeek === 0 || dayOfWeek === 6) penalty += 0.2; + } + return penalty; +} + +function codeReviewPenalty(request: SessionRequest): number { + return request.aiCode.length > 0 && request.totalElapsed != null && request.totalElapsed < 5000 ? 0.5 : 0; +} + +function toolMasteryPenalty(request: SessionRequest): number { + return request.toolsUsed.length === 0 && request.slashCommand === '' ? 0.3 : 0; +} + +function requestPenalty(request: SessionRequest, group: PracticeGroup): number { + if (group === 'prompt-quality') return promptQualityPenalty(request); + if (group === 'session-hygiene') return sessionHygienePenalty(request); + if (group === 'code-review') return codeReviewPenalty(request); + if (group === 'tool-mastery') return toolMasteryPenalty(request); + return 0; +} + +function weekScore(weekRequests: SessionRequest[], group: PracticeGroup): number { + const total = weekRequests.length; + if (total === 0) return 100; + const badCount = weekRequests.reduce((sum, request) => sum + requestPenalty(request, group), 0); + const rate = badCount / total; + return Math.max(0, Math.round(100 - rate * 100)); +} + +export function computeWeeklyTrend(reqs: SessionRequest[]): { labels: string[]; counts: number[] } { + const weekCounts = new Map(); + for (const r of reqs) { + if (!r.timestamp) continue; + const d = new Date(r.timestamp); + const week = isoWeek(d); + let count = 0; + if (r.isCanceled) count++; + if (r.messageLength < 30 && r.messageLength > 0) count++; + if (r.referencedFiles.length === 0 && !(r.variableKinds['file'] > 0) && r.editedFiles.length === 0) count++; + if (r.timestamp && new Date(r.timestamp).getHours() < 5) count++; + const dow = new Date(r.timestamp).getDay(); + if (dow === 0 || dow === 6) count++; + if (count > 0) weekCounts.set(week, (weekCounts.get(week) || 0) + count); + } + const sortedWeeks = Array.from(weekCounts.keys()).sort(); + return { + labels: sortedWeeks, + counts: sortedWeeks.map(w => weekCounts.get(w) || 0), + }; +} + +export function computeWeeklyScores(reqs: SessionRequest[]): { labels: string[]; series: { group: PracticeGroup; scores: number[] }[] } { + const allGroups: PracticeGroup[] = ['prompt-quality', 'session-hygiene', 'code-review', 'tool-mastery']; + const weekReqs = new Map(); + + for (const r of reqs) { + if (!r.timestamp) continue; + const week = isoWeek(new Date(r.timestamp)); + if (!weekReqs.has(week)) weekReqs.set(week, []); + weekReqs.get(week)!.push(r); + } + + const sortedWeeks = Array.from(weekReqs.keys()).sort(); + + return { + labels: sortedWeeks, + series: allGroups.map(group => ({ + group, + scores: sortedWeeks.map(w => weekScore(weekReqs.get(w)!, group)), + })), + }; +} diff --git a/packages/ai-coach-analytics/src/dsl/index.ts b/packages/ai-coach-analytics/src/dsl/index.ts new file mode 100644 index 0000000..7a2a9c6 --- /dev/null +++ b/packages/ai-coach-analytics/src/dsl/index.ts @@ -0,0 +1,227 @@ +/*--------------------------------------------------------------------------------------------- + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the MIT License. See LICENSE in the project root for license information. + *--------------------------------------------------------------------------------------------*/ + +/** + * Public API for the metric expression DSL. + * + * Compiles source expressions into executable functions. + * Re-exports schema and primitives for discovery UIs. + */ + +import { lex } from './lexer'; +import { parse } from './parser'; +import { evaluate } from './interpreter'; +import type { FilterFn, TriggerFn, MetricAggregation } from './types'; + +export { FIELD_SCHEMA, METRIC_PRIMITIVES, FUNCTION_CATALOG } from './schema'; +export type { FieldInfo, MetricPrimitive, FunctionInfo } from './schema'; +export type { ASTNode, FilterFn, TriggerFn, MetricAggregation } from './types'; + +/** + * Evaluate a DSL expression string against a context object. + * Returns the raw result (number, string, boolean, object, array, etc.). + */ +export function evaluateExpression(expr: string, ctx: Record): unknown { + const trimmed = expr.trim(); + if (!trimmed) return null; + try { + const tokens = lex(trimmed); + const ast = parse(tokens); + return evaluate(ast, ctx); + } catch { + return null; + } +} + +/** + * Compile a filter expression string into a predicate function. + * The function takes a single row (request or session) as a flat object + * and returns true if the row matches the filter. + */ +export function compileFilter(expr: string): FilterFn { + const trimmed = expr.trim(); + if (!trimmed) return () => true; + const tokens = lex(trimmed); + const ast = parse(tokens); + return (row: Record) => { + try { + return !!evaluate(ast, row); + } catch { + return false; + } + }; +} + +/** + * Compile a trigger expression string into a predicate. + * Evaluated against the emission object (count, total, ratio, extra.*). + */ +export function compileTrigger(expr: string): TriggerFn { + const trimmed = expr.trim(); + if (!trimmed) return () => false; + const tokens = lex(trimmed); + const ast = parse(tokens); + return (emission) => { + try { + const ctx: Record = { + count: emission.count, + total: emission.total, + ratio: emission.ratio, + ...emission.extra, + extra: emission.extra, + }; + return !!evaluate(ast, ctx); + } catch { + return false; + } + }; +} + +/** + * Parse an aggregation expression like "ratio", "count", "avg(totalElapsed)". + */ +export function parseAggregation(expr: string): MetricAggregation { + const trimmed = expr.trim(); + const lower = trimmed.toLowerCase(); + + if (lower === 'ratio') return { type: 'ratio' }; + if (lower === 'count') return { type: 'count' }; + + const funcMatch = trimmed.match(/^(sum|avg|min|max|unique|percentile)\((\w[\w.]*)\s*(?:,\s*(\d+))?\)$/i); + if (funcMatch) { + const result: MetricAggregation = { + type: funcMatch[1].toLowerCase() as MetricAggregation['type'], + field: funcMatch[2], + }; + if (funcMatch[3]) result.percentile = Number.parseInt(funcMatch[3]); + return result; + } + + // Default to count + return { type: 'count' }; +} + +/** + * Compute an aggregation over filtered rows. + */ +export function computeAggregation( + agg: MetricAggregation, + matchedRows: Record[], + totalRows: Record[], +): number { + switch (agg.type) { + case 'count': + return matchedRows.length; + case 'ratio': + return totalRows.length > 0 ? matchedRows.length / totalRows.length : 0; + case 'sum': + return matchedRows.reduce((s, r) => s + toNum(resolveField(r, agg.field!)), 0); + case 'avg': { + if (matchedRows.length === 0) return 0; + const total = matchedRows.reduce((s, r) => s + toNum(resolveField(r, agg.field!)), 0); + return total / matchedRows.length; + } + case 'min': + return matchedRows.reduce((m, r) => Math.min(m, toNum(resolveField(r, agg.field!))), Infinity); + case 'max': + return matchedRows.reduce((m, r) => Math.max(m, toNum(resolveField(r, agg.field!))), -Infinity); + case 'unique': + return new Set(matchedRows.map(r => resolveField(r, agg.field!))).size; + case 'percentile': { + const values = matchedRows.map(r => toNum(resolveField(r, agg.field!))).sort((a, b) => a - b); + if (values.length === 0) return 0; + const p = (agg.percentile ?? 50) / 100; + const idx = Math.floor(p * (values.length - 1)); + return values[idx]; + } + } +} + +/** + * Evaluate a format/example template string against a row. + * Handles {{field}}, {{field | filter:arg}}, and nested {{field.sub}}. + */ +export function evaluateTemplate(template: string, row: Record): string { + return template.replaceAll(/\{\{(.+?)\}\}/g, (_match, rawExpr: string) => { + try { + // Support pipe filters: {{expr | filter:arg}} + const pipeIdx = rawExpr.indexOf('|'); + let expr = rawExpr.trim(); + let filterName = ''; + let filterArg = ''; + if (pipeIdx >= 0) { + expr = rawExpr.substring(0, pipeIdx).trim(); + const filterPart = rawExpr.substring(pipeIdx + 1).trim(); + const colonIdx = filterPart.indexOf(':'); + if (colonIdx >= 0) { + filterName = filterPart.substring(0, colonIdx).trim(); + filterArg = filterPart.substring(colonIdx + 1).trim(); + } else { + filterName = filterPart; + } + } + + const tokens = lex(expr); + const ast = parse(tokens); + let result = evaluate(ast, row); + if (result == null) return ''; + + // Apply filter + if (filterName === 'truncate' && typeof result === 'string') { + const maxLen = Number.parseInt(filterArg, 10) || 80; + if (result.length > maxLen) { + result = result.substring(0, maxLen) + '...'; + } + } else if (filterName === 'clip' && typeof result === 'string') { + // Always clip to N chars and append "..." (even if string is shorter) + const maxLen = Number.parseInt(filterArg, 10) || 80; + result = result.substring(0, maxLen) + '...'; + } + + return String(result); + } catch { + return `{{${rawExpr}}}`; + } + }); +} + +/** + * Try to compile an expression and return any error message, or null if valid. + */ +export function validateExpression(expr: string): string | null { + try { + const tokens = lex(expr.trim()); + parse(tokens); + return null; + } catch (e) { + return e instanceof Error ? e.message : String(e); + } +} + +/* ---- Internal helpers ---- */ + +function resolveField(row: Record, field: string): unknown { + const parts = field.split('.'); + let current: unknown = row; + for (const part of parts) { + if (current == null) return undefined; + if (typeof current === 'object' && !Array.isArray(current)) { + current = (current as Record)[part]; + } else if (Array.isArray(current) && part === 'length') { + return current.length; + } else if (typeof current === 'string' && part === 'length') { + return current.length; + } else { + return undefined; + } + } + return current; +} + +function toNum(v: unknown): number { + if (typeof v === 'number') return v; + if (typeof v === 'string') { const n = Number.parseFloat(v); return Number.isNaN(n) ? 0 : n; } + return 0; +} diff --git a/packages/ai-coach-analytics/src/dsl/interpreter.ts b/packages/ai-coach-analytics/src/dsl/interpreter.ts new file mode 100644 index 0000000..2858a6e --- /dev/null +++ b/packages/ai-coach-analytics/src/dsl/interpreter.ts @@ -0,0 +1,1785 @@ +/*--------------------------------------------------------------------------------------------- + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the MIT License. See LICENSE in the project root for license information. + *--------------------------------------------------------------------------------------------*/ + +/** + * AST interpreter for the metric expression DSL. + * + * Evaluates an AST node against a context object (a row of data, or an + * emission object). Returns a JS value (number, string, boolean, array). + * + * Built-in functions available inside expressions: + * - length(x) array or string length + * - contains(s, sub) string contains + * - startsWith(s, sub) string starts with + * - endsWith(s, sub) string ends with + * - matches(s, /re/) regex match + * - lower(s) to lower case + * - upper(s) to upper case + * - trim(s) strip whitespace + * - abs(n) absolute value + * - floor(n) floor + * - ceil(n) ceiling + * - round(n) round + * - min(a, b) minimum + * - max(a, b) maximum + * - hour(ts) 0-23 from epoch ms + * - dayOfWeek(ts) 0=Sun 6=Sat from epoch ms + * - includes(arr, val) array includes value + * - some(arr, field) true if any element has truthy field + * - count(arr) array length + * - sum(arr, field) sum a numeric field across array elements + * - avg(arr, field) average a numeric field + * - unique(arr, field) count distinct values of field + * - truncate(s, n) truncate string to n chars + * - substr(s, start, len) substring + * - split(s, sep) split string into array + * - join(arr, sep) join array into string + * - keys(obj) object keys + * - values(obj) object values + * - has(obj, key) check if object has key + * - coalesce(a, b, ...) first non-null value + * + * Aggregate functions (operate on arrays of objects): + * - first(arr) first element + * - at(arr, idx) element at index + * - substring(s, start, end) substring + * - lineCount(s) non-empty line count + * - isStructured(s) has bullets/numbers/headings + * - hasProfanity(s) profanity detection + * - normalizeModel(id) normalize model name + * - sumAiLoc(reqs) sum aiCode.loc across requests + * - workTypeCount(reqs) distinct work types in requests + * - hasReviewFollowup(reqs) any follow-up mentions review keywords + * - groupTopKey(arr, field) most frequent value key + * - groupTopCount(arr, field) most frequent value count + * - groupTopShare(arr, field) proportion of most frequent value + * - countWhere(arr, field, op, value) count matching rows + * - sumField(arr, field) sum numeric field across rows + * - avgField(arr, field) average numeric field across rows + * - flatCount(arr, field) sum of sub-array lengths + * - flatSomeWhere(arr, field, subField, value) any sub-element matches + * - adjacentPairCount(sessions, minLoc, maxGapMs) speed-accept pairs + * - flowScoreStats(sessions, minTimedReqs, rapidMs) flow-state analysis + * - langExplorationWeeks(reqs) language exploration stats + * - mdRatioByWorkspace(sessions, minLoc, docLangs) markdown ratio per workspace + * - devcontainerStats(sessions, reqs) devcontainer usage stats + * - contextGapCount(reqs) context engineering gap analysis + * - yoloStats(reqs) yolo-mode tool confirmation stats + * - reasoningEffortStats(reqs, premiumLevel) ratio of high/max effort use + * - instructionBloatStats(sessions, maxBytes) custom-instructions size analysis + * - depRetryStats(sessions, reqs) repeated dep-install attempts in VS Code terminal + * - excessFileContextStats(reqs, minFiles) outliers attaching huge file contexts + * - hasSkillByPattern(reqs, pattern) any skillsUsed entry matches a regex + */ + +import { containsProfanity as _containsProfanity, extractProfaneWords } from '../profanity'; +import { compileSafe, testSafe } from './safe-regex'; +import { ASTNode } from './types'; + +export class InterpreterError extends Error { + constructor(message: string) { + super(`DSL runtime error: ${message}`); + } +} + +/** Safely convert an unknown value to string (never returns '[object Object]'). */ +function asStr(v: unknown, fallback = ''): string { + if (typeof v === 'string') return v; + if (typeof v === 'number' || typeof v === 'boolean') return String(v); + return fallback; +} + +/** Return arg as number if non-null, else return def. */ +function numOpt(arg: unknown, def: number): number { + return arg != null ? toNum(arg) : def; +} + +/** Hard ceiling on AST-node evaluations per top-level `evaluate()` call + * (including pipe iterations over data rows). Real expressions stay far + * below this; only a pathological rule file can hit it. The throw is caught + * by the compiled-expression wrappers in dsl/index.ts, so an offending rule + * degrades to no-match instead of stalling analysis. */ +const MAX_EVAL_STEPS = 5_000_000; +/** Recursion guard: parse depth is capped at 64, but calls and pipes add frames. */ +const MAX_EVAL_DEPTH = 500; +let evalSteps = 0; +let evalDepth = 0; + +export function evaluate(node: ASTNode, ctx: Record): unknown { + if (evalDepth === 0) evalSteps = 0; + if (++evalSteps > MAX_EVAL_STEPS) { + throw new InterpreterError(`evaluation budget exceeded (${MAX_EVAL_STEPS} steps)`); + } + if (evalDepth >= MAX_EVAL_DEPTH) { + throw new InterpreterError('expression nesting too deep'); + } + evalDepth++; + try { + return evaluateNode(node, ctx); + } finally { + evalDepth--; + } +} + +function evaluateNode(node: ASTNode, ctx: Record): unknown { + switch (node.type) { + case 'number': + case 'string': + case 'boolean': + return node.value; + + case 'identifier': + return resolveIdentifier(node.name, ctx); + + case 'field_access': + return evaluateFieldAccess(node, ctx); + + case 'binary': + return evaluateBinary(node.op, node.left, node.right, ctx); + + case 'unary': + if (node.op === 'NOT') return !toBool(evaluate(node.operand, ctx)); + throw new InterpreterError(`Unknown unary operator: ${node.op}`); + + case 'call': + return evaluateCall(node.name, node.args, ctx); + + case 'pipe': + return evaluatePipe(node.value, node.filter, node.arg, ctx); + + case 'array': + return node.elements.map(el => evaluate(el, ctx)); + } +} + +function evaluateFieldAccess(node: ASTNode & { type: 'field_access' }, ctx: Record): unknown { + const obj = evaluate(node.object, ctx); + if (obj == null) return undefined; + if (typeof obj === 'object') { + const field = node.field; + if (field === '__proto__' || field === 'constructor' || field === 'prototype') return undefined; + return Object.hasOwn(obj as Record, field) + ? (obj as Record)[field] + : undefined; + } + if (node.field === 'length' && (typeof obj === 'string' || Array.isArray(obj))) { + return (obj as string | unknown[]).length; + } + return undefined; +} + +function resolveIdentifier(name: string, ctx: Record): unknown { + return Object.hasOwn(ctx, name) ? ctx[name] : undefined; +} + +function evaluateBinary(op: string, left: ASTNode, right: ASTNode, ctx: Record): unknown { + // Short-circuit logical operators + if (op === 'AND') return toBool(evaluate(left, ctx)) && toBool(evaluate(right, ctx)); + if (op === 'OR') return toBool(evaluate(left, ctx)) || toBool(evaluate(right, ctx)); + + const lv = evaluate(left, ctx); + const rv = evaluate(right, ctx); + + switch (op) { + case '<': return toNum(lv) < toNum(rv); + case '>': return toNum(lv) > toNum(rv); + case '<=': return toNum(lv) <= toNum(rv); + case '>=': return toNum(lv) >= toNum(rv); + case '==': return looseEquals(lv, rv); + case '!=': return !looseEquals(lv, rv); + // Arithmetic operators + case '+': return toNum(lv) + toNum(rv); + case '-': return toNum(lv) - toNum(rv); + case '*': return toNum(lv) * toNum(rv); + case '/': { const d = toNum(rv); return d === 0 ? 0 : toNum(lv) / d; } + default: throw new InterpreterError(`Unknown operator: ${op}`); + } +} + +function evaluateCall(name: string, argNodes: ASTNode[], ctx: Record): unknown { + const handler = (CALL_DISPATCH as Record) => unknown>)[name]; + if (!handler) throw new InterpreterError(`Unknown function: ${name}`); + return handler(argNodes, ctx); +} + + +function evaluatePipe(valueNode: ASTNode, filter: string, argNode: ASTNode | undefined, ctx: Record): unknown { + const value = evaluate(valueNode, ctx); + const arg = argNode ? evaluate(argNode, ctx) : undefined; + + switch (filter) { + case 'truncate': { + const s = asStr(value); + const n = numOpt(arg, 80); + return s.length > n ? s.substring(0, n) + '...' : s; + } + case 'lower': return asStr(value).toLowerCase(); + case 'upper': return asStr(value).toUpperCase(); + case 'trim': return asStr(value).trim(); + case 'round': return Math.round(toNum(value)); + case 'floor': return Math.floor(toNum(value)); + case 'ceil': return Math.ceil(toNum(value)); + case 'abs': return Math.abs(toNum(value)); + case 'pct': return `${(toNum(value) * 100).toFixed(numOpt(arg, 0))}%`; + case 'fixed': return toNum(value).toFixed(numOpt(arg, 2)); + case 'locale': return toNum(value).toLocaleString(); + default: + throw new InterpreterError(`Unknown pipe filter: ${filter}`); + } +} + +/* ---- Type coercion helpers ---- */ + +function toNum(v: unknown): number { + if (typeof v === 'number') return v; + if (typeof v === 'boolean') return v ? 1 : 0; + if (typeof v === 'string') { + const n = Number.parseFloat(v); + return Number.isNaN(n) ? 0 : n; + } + if (v == null) return 0; + return 0; +} + +function toBool(v: unknown): boolean { + if (typeof v === 'boolean') return v; + if (typeof v === 'number') return v !== 0; + if (typeof v === 'string') return v.length > 0; + if (v == null) return false; + if (Array.isArray(v)) return v.length > 0; + return true; +} + +function looseEquals(a: unknown, b: unknown): boolean { + if (a === b) return true; + if (a == null && b == null) return true; + if (typeof a === 'number' && typeof b === 'string') return a === Number.parseFloat(b); + if (typeof a === 'string' && typeof b === 'number') return Number.parseFloat(a) === b; + if (typeof a === 'boolean' && typeof b === 'number') return (a ? 1 : 0) === b; + if (typeof a === 'number' && typeof b === 'boolean') return a === (b ? 1 : 0); + return false; +} + +/* ── Model tier lookup (mirrors helpers.ts MODEL_MULTIPLIERS) ── */ + +const MODEL_TIERS: Record = { + 'claude-opus-4.7': 7.5, + 'claude-opus-4.6-fast': 30, 'claude-opus-4.6': 3, 'claude-opus-4.5': 3, + 'claude-haiku-4.5': 0.33, + 'claude-sonnet-4.6': 1, 'claude-sonnet-4.5': 1, 'claude-sonnet-4': 1, + 'claude-opus-41': 1, 'claude-opus-4': 1, + 'claude-3.5-sonnet': 1, 'claude-3.7-sonnet': 1, 'claude-4': 1, 'claude-3-opus': 3, + 'gpt-5.4-mini': 0.33, 'gpt-5.4-nano': 0.2, 'gpt-5.4': 1, + 'gpt-5.3-codex': 1, 'gpt-5.2-codex': 1, 'gpt-5.2': 1, + 'gpt-5.1-codex-max': 1, 'gpt-5.1-codex-mini': 0.33, 'gpt-5.1-codex': 1, 'gpt-5.1': 1, + 'gpt-5.5': 3, 'gpt-5-mini': 0.33, + 'o4-mini': 2, 'o3-mini': 1, 'o3': 3, 'o1-mini': 1, 'o1-preview': 2, 'o1': 2, + 'gpt-4.1-nano': 0.2, 'gpt-4.1-mini': 0.5, 'gpt-4.1': 1, + 'gpt-4-turbo': 1, 'gpt-4': 1, + 'gemini-3.1-pro': 1, 'gemini-3-pro': 1, 'gemini-3-flash': 0.33, + 'gemini-2.5-pro': 1, 'gemini-2.0-flash': 0.3, + 'grok-code-fast-1': 0.25, +}; + +function modelTierLookup(raw: string): number { + const id = raw.replace(/^(openai\/|anthropic\/|google\/)/, '').replace(/-\d{4}-\d{2}-\d{2}$/, '').toLowerCase(); + for (const [k, v] of Object.entries(MODEL_TIERS)) { + if (id.includes(k)) return v; + } + return 0; +} + +/* ── Work type classification (mirrors helpers.ts classifyWorkType) ── */ + +/** Extract inline regex flags like (?i), (?im), (?ims) from the start of a pattern */ +function parseRegexFlags(pattern: string): { pattern: string; flags: string } { + const m = pattern.match(/^\(\?([gimsuy]+)\)/); + if (m) return { pattern: pattern.substring(m[0].length), flags: m[1] }; + return { pattern, flags: '' }; +} + +const WORK_TYPE_PATTERNS: [RegExp, string][] = [ + [/\b(bug|fix|error|issue|crash|broken|wrong|fail|debug)\b/i, 'bug fix'], + [/\b(refactor|clean ?up|rename|restructure|reorganize|simplify)\b/i, 'refactor'], + [/\b(test|spec|coverage|assert|expect|mock|stub)\b/i, 'test'], + [/\b(doc|readme|comment|jsdoc|typedoc|explain)\b/i, 'documentation'], + [/\b(deploy|ci|cd|pipeline|docker|kubernetes|helm|terraform|infra)\b/i, 'devops'], + [/\b(style|css|layout|design|ui|ux|theme|color|font)\b/i, 'styling'], + [/\b(config|setup|install|init|bootstrap|scaffold)\b/i, 'configuration'], + [/\b(perf|optim|speed|cache|memory|benchmark)\b/i, 'performance'], + [/\b(security|auth|permission|encrypt|token|oauth|cors)\b/i, 'security'], + [/\b(migration?|upgrade|update|version|deprecat)\b/i, 'migration'], +]; + +function classifyWorkText(text: string): string { + const sample = text.length > 300 ? text.substring(0, 300) : text; + for (const [re, type] of WORK_TYPE_PATTERNS) { + if (re.test(sample)) return type; + } + return 'feature'; +} + +/* ── Resolve nested field path ── */ + +function resolveField(row: Record, field: string): unknown { + const parts = field.split('.'); + let current: unknown = row; + for (const part of parts) { + if (current == null) return undefined; + if (typeof current === 'object' && !Array.isArray(current)) { + current = (current as Record)[part]; + } else if (Array.isArray(current) && part === 'length') { + return current.length; + } else { + return undefined; + } + } + return current; +} + +/* ── GroupBy helper ── */ + +function groupTop(arr: unknown[], fieldName: string): { key: string; count: number; share: number } { + const counts = new Map(); + for (const item of arr) { + const val = typeof item === 'object' && item + ? asStr(resolveField(item as Record, fieldName)) + : ''; + counts.set(val, (counts.get(val) || 0) + 1); + } + let topKey = '', topCount = 0; + for (const [k, c] of counts) { + if (c > topCount) { topKey = k; topCount = c; } + } + return { key: topKey, count: topCount, share: arr.length > 0 ? topCount / arr.length : 0 }; +} + +/* ── Normalize model ID ── */ + +function normalizeModelId(raw: string): string { + return raw.replace(/^(openai\/|anthropic\/|google\/)/, '').replace(/-\d{4}-\d{2}-\d{2}$/, '').toLowerCase().trim() || 'untracked'; +} + +/* ── Profanity detection ── */ +// Shared implementation lives in ../profanity.ts + +function detectProfanity(text: string): boolean { + return _containsProfanity(text); +} + +/* ── Speed-accept pairs (adjacent request analysis) ── */ + +function computeSpeedAcceptPairs(sessions: Record[], minLoc: number, maxGap: number): { count: number; avgLoc: number; avgGap: number } { + let count = 0; + let totalLoc = 0; + let totalGap = 0; + for (const session of sessions) { + const reqs = session.requests; + if (!Array.isArray(reqs) || reqs.length < 2) continue; + for (let i = 0; i < reqs.length - 1; i++) { + const prev = reqs[i] as Record; + const next = reqs[i + 1] as Record; + // Sum AI LOC from prev + const aiCode = prev.aiCode; + if (!Array.isArray(aiCode)) continue; + const aiLoc = aiCode.reduce((s: number, b: unknown) => s + toNum((b as Record)?.loc), 0); + if (aiLoc < minLoc) continue; + // Compute gap + const prevEnd = toNum(prev.timestamp) + toNum(prev.totalElapsed); + const nextStart = toNum(next.timestamp); + if (prevEnd <= 0 || nextStart <= 0) continue; + const gap = nextStart - prevEnd; + if (gap >= 0 && gap <= maxGap) { + count++; + totalLoc += aiLoc; + totalGap += gap; + } + } + } + return { + count, + avgLoc: count > 0 ? Math.round(totalLoc / count) : 0, + avgGap: count > 0 ? Math.round(totalGap / count / 1000) : 0, + }; +} + +/* ── Flow-state scoring ── */ + +function computeFlowScoreStats( + sessions: Record[], + minTimedReqs: number, + rapidMs: number, +): { fragmentedDays: number; totalDays: number; avgScore: number; lowScoreRate: number } { + const dayScores = new Map(); + + for (const session of sessions) { + const reqs = session.requests; + if (!Array.isArray(reqs)) continue; + const timed = (reqs as Record[]).filter( + r => toNum(r.timestamp) > 0 && toNum(r.totalElapsed) > 0 + ); + if (timed.length < minTimedReqs) continue; + + // Sort by timestamp + timed.sort((a, b) => toNum(a.timestamp) - toNum(b.timestamp)); + + // Compute inter-request gaps + const gaps: number[] = []; + for (let i = 1; i < timed.length; i++) { + const prevEnd = toNum(timed[i - 1].timestamp) + toNum(timed[i - 1].totalElapsed); + const nextStart = toNum(timed[i].timestamp); + gaps.push(Math.max(0, nextStart - prevEnd)); + } + if (gaps.length === 0) continue; + + // Median gap + const sorted = [...gaps].sort((a, b) => a - b); + const median = sorted[Math.floor(sorted.length / 2)]; + + // Flow score: rapid-followup rate + latency bonus + const rapidCount = gaps.filter(g => g <= rapidMs).length; + const rapidRate = rapidCount / gaps.length; + const latencyScore = Math.max(0, 100 - (median / 1000)); // penalty per second + const score = Math.round(rapidRate * 60 + Math.min(latencyScore, 40)); + + // Group by day + const day = new Date(toNum(timed[0].timestamp)).toISOString().substring(0, 10); + if (!dayScores.has(day)) dayScores.set(day, []); + dayScores.get(day)!.push(score); + } + + const totalDays = dayScores.size; + if (totalDays === 0) return { fragmentedDays: 0, totalDays: 0, avgScore: 100, lowScoreRate: 0 }; + + let fragmentedDays = 0; + let totalScore = 0; + let scoreCount = 0; + for (const scores of dayScores.values()) { + const avg = scores.reduce((a, b) => a + b, 0) / scores.length; + totalScore += avg; + scoreCount++; + if (avg < 50) fragmentedDays++; + } + + return { + fragmentedDays, + totalDays, + avgScore: scoreCount > 0 ? Math.round(totalScore / scoreCount) : 100, + lowScoreRate: totalDays > 0 ? fragmentedDays / totalDays : 0, + }; +} + +/* ── Language exploration tracking ── */ + +function computeLangExploration(reqs: Record[]): { weeksSinceNew: number; totalLangs: number; recentNew: number; totalWeeks: number } { + const IGNORE = new Set(['text', 'plaintext', 'unknown', 'json', 'yaml', 'toml', 'xml', 'csv', 'ini', 'env', 'markdown', 'md']); + const weekLangs = new Map>(); + + for (const r of reqs) { + const ts = toNum(r.timestamp); + if (ts <= 0) continue; + const d = new Date(ts); + const week = `${d.getFullYear()}-W${String(Math.ceil((d.getDate() + new Date(d.getFullYear(), d.getMonth(), 1).getDay()) / 7)).padStart(2, '0')}`; + if (!weekLangs.has(week)) weekLangs.set(week, new Set()); + const set = weekLangs.get(week)!; + for (const field of ['aiCode', 'userCode'] as const) { + const blocks = r[field]; + if (!Array.isArray(blocks)) continue; + for (const b of blocks) { + const lang = asStr((b as Record)?.language).toLowerCase(); + if (lang && !IGNORE.has(lang)) set.add(lang); + } + } + } + + const weeks = [...weekLangs.keys()].sort(); + if (weeks.length === 0) return { weeksSinceNew: 0, totalLangs: 0, recentNew: 0, totalWeeks: 0 }; + + const seen = new Set(); + let lastNewWeek = 0; + for (let i = 0; i < weeks.length; i++) { + const langs = weekLangs.get(weeks[i])!; + let isNew = false; + for (const l of langs) { + if (!seen.has(l)) { seen.add(l); isNew = true; } + } + if (isNew) lastNewWeek = i; + } + + const weeksSinceNew = weeks.length - 1 - lastNewWeek; + const recentNew = weeksSinceNew === 0 ? 1 : 0; + return { weeksSinceNew, totalLangs: seen.size, recentNew, totalWeeks: weeks.length }; +} + +/* ── Markdown ratio by workspace ── */ + +function computeMdRatio( + sessions: Record[], + minLoc: number, + docLangs: string[], +): { + lowCount: number; + totalWorkspaces: number; + overallRatio: number; + totalCodeLoc: number; + totalMdLoc: number; + workspaces: { name: string; codeLoc: number; mdLoc: number; ratio: number; isLow: boolean }[]; +} { + const docSet = new Set(docLangs.map(l => l.toLowerCase())); + const workspace = new Map(); + + for (const session of sessions) { + const ws = asStr(session.workspaceName) || asStr(session.workspaceId) || 'unknown'; + if (!workspace.has(ws)) workspace.set(ws, { md: 0, code: 0 }); + const stats = workspace.get(ws)!; + const reqs = session.requests; + if (!Array.isArray(reqs)) continue; + for (const r of reqs as Record[]) { + const aiCode = r.aiCode; + if (!Array.isArray(aiCode)) continue; + for (const b of aiCode as Record[]) { + const lang = asStr(b.language).toLowerCase(); + const loc = toNum(b.loc); + if (docSet.has(lang)) stats.md += loc; + else stats.code += loc; + } + } + } + + let lowCount = 0; + let totalMd = 0; + let totalCode = 0; + const workspaces: { name: string; codeLoc: number; mdLoc: number; ratio: number; isLow: boolean }[] = []; + for (const [name, stats] of workspace.entries()) { + const total = stats.md + stats.code; + totalMd += stats.md; + totalCode += stats.code; + const ratio = total > 0 ? stats.md / total : 0; + const isLow = total >= minLoc && ratio < 0.05; + if (isLow) lowCount++; + workspaces.push({ name, codeLoc: stats.code, mdLoc: stats.md, ratio, isLow }); + } + // Sort: low-ratio workspaces first, then by code volume descending + workspaces.sort((a, b) => (a.isLow === b.isLow ? b.codeLoc - a.codeLoc : a.isLow ? -1 : 1)); + + return { + lowCount, + totalWorkspaces: workspace.size, + overallRatio: (totalMd + totalCode) > 0 ? totalMd / (totalMd + totalCode) : 0, + totalCodeLoc: totalCode, + totalMdLoc: totalMd, + workspaces, + }; +} + +/* ── Devcontainer stats ── */ + +const VSCODE_HARNESSES = new Set(['VS Code', 'VS Code Insiders', 'Local Agent', 'Local Agent (Insiders)']); + +function computeDevcontainerStats( + sessions: Record[], + reqs: Record[], +): { terminalReqs: number; vscodeReqs: number; sandboxedTerminalReqs: number; totalTerminalReqs: number; terminalRate: number; vscodeSessionCount: number } { + const vscodeSessions = sessions.filter(s => VSCODE_HARNESSES.has(asStr(s.harness))); + const sessionIsContained = new Map(); + for (const s of vscodeSessions) { + sessionIsContained.set(String(s.sessionId), s.hasDevcontainer === true); + } + const vscodeReqs = reqs.filter(r => sessionIsContained.has(asStr(r.sessionId))); + + let hostedTerminalReqs = 0; + let sandboxedTerminalReqs = 0; + for (const r of vscodeReqs) { + const confs = r.toolConfirmations; + if (!Array.isArray(confs) || !confs.some((tc: unknown) => (tc as Record).isTerminal)) continue; + if (sessionIsContained.get(asStr(r.sessionId)) === true) sandboxedTerminalReqs++; + else hostedTerminalReqs++; + } + const totalTerminalReqs = hostedTerminalReqs + sandboxedTerminalReqs; + + return { + terminalReqs: hostedTerminalReqs, + vscodeReqs: vscodeReqs.length, + sandboxedTerminalReqs, + totalTerminalReqs, + terminalRate: vscodeReqs.length > 0 ? hostedTerminalReqs / vscodeReqs.length : 0, + vscodeSessionCount: vscodeSessions.length, + }; +} + +/* ── Context engineering gaps ── */ + +function computeContextGaps(reqs: Record[]): { gapCount: number; gaps: string[]; reqCount: number } { + const gaps: string[] = []; + const total = reqs.length; + + // 1. Sub-agent / multi-agent usage + const hasSubAgents = reqs.some(r => { + const name = asStr(r.agentName); + return name && name !== 'copilot' && asStr(r.agentMode) === 'agent'; + }); + if (!hasSubAgents) gaps.push('No sub-agent usage detected'); + + // 2. Skills usage + const hasSkills = reqs.some(r => { + const skills = r.skillsUsed; + return Array.isArray(skills) && skills.length > 0; + }); + if (!hasSkills) gaps.push('No skills used in any request'); + + // 3. MCP tools + const hasMcp = reqs.some(r => { + const tools = r.toolsUsed; + return Array.isArray(tools) && tools.some((t: unknown) => String(t).startsWith('mcp_')); + }); + if (!hasMcp) gaps.push('No MCP tool integration'); + + // 4. File reference rate + const fileRefCount = reqs.filter(r => { + const refs = r.referencedFiles; + return Array.isArray(refs) && refs.length > 0; + }).length; + if (total > 0 && fileRefCount / total < 0.1) gaps.push('Low file reference rate (< 10%)'); + + // 5. Custom instructions rate + const instrCount = reqs.filter(r => { + const instr = r.customInstructions; + return Array.isArray(instr) && instr.length > 0; + }).length; + if (total > 0 && instrCount / total < 0.05) gaps.push('Low custom instructions usage (< 5%)'); + + return { gapCount: gaps.length, gaps, reqCount: total }; +} + +/* ── Yolo-mode stats ── */ + +function computeYoloStats(reqs: Record[]): { autoApproved: number; totalConfirmations: number; ratio: number; withConfirmationsCount: number } { + let totalConf = 0; + let autoApproved = 0; + let withConf = 0; + for (const r of reqs) { + const confs = r.toolConfirmations; + if (!Array.isArray(confs) || confs.length === 0) continue; + withConf++; + for (const tc of confs as Record[]) { + totalConf++; + const scope = asStr(tc.autoApproveScope); + if (scope === 'session' || scope === 'always') autoApproved++; + } + } + return { + autoApproved, + totalConfirmations: totalConf, + ratio: totalConf > 0 ? autoApproved / totalConf : 0, + withConfirmationsCount: withConf, + }; +} + +/* ── Reasoning effort overuse ── */ + +function computeReasoningEffortStats( + reqs: Record[], + premiumLevel: string, +): { premiumCount: number; totalKnown: number; total: number; ratio: number; topTier: string } { + const premiumSet = premiumLevel === 'max' + ? new Set(['max']) + : new Set(['high', 'max']); + let premiumCount = 0; + let totalKnown = 0; + const tierCounts = new Map(); + for (const r of reqs) { + const eff = r.reasoningEffort; + if (eff === undefined || eff === null) continue; + const tier = asStr(eff).toLowerCase(); + if (!['max', 'high', 'medium', 'low'].includes(tier)) continue; + totalKnown++; + tierCounts.set(tier, (tierCounts.get(tier) ?? 0) + 1); + if (premiumSet.has(tier)) premiumCount++; + } + let topTier = ''; + let topCount = -1; + for (const [t, c] of tierCounts.entries()) { + if (c > topCount) { topTier = t; topCount = c; } + } + return { + premiumCount, + totalKnown, + total: reqs.length, + ratio: totalKnown > 0 ? premiumCount / totalKnown : 0, + topTier, + }; +} + +/* ── Instruction-bloat (oversized .github/copilot-instructions.md) ── */ + +function computeInstructionBloatStats( + sessions: Record[], + maxBytes: number, +): { bloatedSessions: number; withInstructionsCount: number; totalSessions: number; maxBytes: number; p95Bytes: number } { + const perWorkspace = new Map(); + for (const s of sessions) { + const ws = asStr(s.workspaceName) || asStr(s.workspaceId); + const bytes = s.customInstructionsBytes; + if (typeof bytes !== 'number' || !Number.isFinite(bytes)) continue; + const prior = perWorkspace.get(ws); + if (prior === undefined || bytes > prior) perWorkspace.set(ws, bytes); + } + let bloated = 0; + let withInstructions = 0; + const sizes: number[] = []; + let topSize = 0; + for (const bytes of perWorkspace.values()) { + sizes.push(bytes); + if (bytes > 0) withInstructions++; + if (bytes > maxBytes) bloated++; + if (bytes > topSize) topSize = bytes; + } + sizes.sort((a, b) => a - b); + const p95 = sizes.length > 0 ? sizes[Math.min(sizes.length - 1, Math.floor(sizes.length * 0.95))] : 0; + return { + bloatedSessions: bloated, + withInstructionsCount: withInstructions, + totalSessions: perWorkspace.size, + maxBytes: topSize, + p95Bytes: p95, + }; +} + +/* ── Excessive file context (huge file attachments per request) ── */ + +function computeExcessFileContextStats( + reqs: Record[], + minFiles: number, +): { outlierCount: number; totalReqs: number; ratio: number; p95Files: number; maxFiles: number } { + let outliers = 0; + let topFiles = 0; + const counts: number[] = []; + for (const r of reqs) { + const refs = r.referencedFiles; + const n = Array.isArray(refs) ? refs.length : 0; + counts.push(n); + if (n > topFiles) topFiles = n; + if (n >= minFiles) outliers++; + } + counts.sort((a, b) => a - b); + const p95 = counts.length > 0 ? counts[Math.min(counts.length - 1, Math.floor(counts.length * 0.95))] : 0; + return { + outlierCount: outliers, + totalReqs: reqs.length, + ratio: reqs.length > 0 ? outliers / reqs.length : 0, + p95Files: p95, + maxFiles: topFiles, + }; +} + +/* ── Duplicate groups (repeated prompts) ── */ + +function computeDuplicateGroups(reqs: Record[], minKeyLen: number, minCount: number): { totalDupes: number; distinctCount: number; topKey: string; topCount: number } { + const map = new Map(); + for (const r of reqs) { + const msg = asStr(r.messageText); + const key = msg.substring(0, 100).toLowerCase().trim(); + if (key.length >= minKeyLen) map.set(key, (map.get(key) || 0) + 1); + } + const dupes = [...map.entries()].filter(([, c]) => c >= minCount); + const totalDupes = dupes.reduce((s, [, c]) => s + c, 0); + dupes.sort((a, b) => b[1] - a[1]); + return { + totalDupes, + distinctCount: dupes.length, + topKey: dupes.length > 0 ? dupes[0][0] : '', + topCount: dupes.length > 0 ? dupes[0][1] : 0, + }; +} + +/* ── Profanity matches across requests ── */ + +function computeProfanityMatches(reqs: Record[]): { count: number; total: number; flaggedWords: string[] } { + let count = 0; + const flagged: string[] = []; + for (const r of reqs) { + const msg = asStr(r.messageText); + const found = extractProfaneWords(msg); + if (found.length > 0) { + count++; + if (flagged.length < 10) flagged.push(...found.slice(0, 3)); + } + } + return { count, total: reqs.length, flaggedWords: [...new Set(flagged)] }; +} + +/* ── Auto-approve stats ── */ + +function computeAutoApproveStats(reqs: Record[]): { + terminalAutoApproved: number; autoApprovedTotal: number; withConfirmations: number; +} { + let withConf = 0, autoTotal = 0, termAuto = 0; + for (const r of reqs) { + const confs = r.toolConfirmations; + if (!Array.isArray(confs) || confs.length === 0) continue; + withConf++; + let hasAutoApprove = false, hasTerminalAutoApprove = false; + for (const tc of confs as Record[]) { + const scope = asStr(tc.autoApproveScope); + if (scope === 'session' || scope === 'always') { + hasAutoApprove = true; + if (tc.isTerminal) hasTerminalAutoApprove = true; + } + } + if (hasAutoApprove) autoTotal++; + if (hasTerminalAutoApprove) termAuto++; + } + return { terminalAutoApproved: termAuto, autoApprovedTotal: autoTotal, withConfirmations: withConf }; +} + +/* ── Group by field and sum another, return top group ── */ + +function groupTopBySum(arr: unknown[], groupField: string, sumField: string): { key: string; sum: number; share: number; count: number } { + const counts = new Map(); + let totalSum = 0; + for (const item of arr) { + if (typeof item !== 'object' || !item) continue; + const row = item as Record; + const key = asStr(resolveField(row, groupField)); + const value = toNum(resolveField(row, sumField)); + counts.set(key, (counts.get(key) || 0) + value); + totalSum += value; + } + let topKey = '', topSum = 0; + for (const [k, s] of counts) { + if (s > topSum) { topKey = k; topSum = s; } + } + return { key: topKey, sum: topSum, share: totalSum > 0 ? topSum / totalSum : 0, count: counts.size }; +} + +/* ── Model overreliance stats ── */ + +function computeModelStats(reqs: Record[]): { topModel: string; topCount: number; topShare: number; modelCount: number; total: number } { + const counts = new Map(); + for (const r of reqs) { + const m = normalizeModelId(asStr(r.modelId, 'untracked')); + counts.set(m, (counts.get(m) || 0) + 1); + } + let topModel = '', topCount = 0; + for (const [k, c] of counts) { + if (c > topCount) { topModel = k; topCount = c; } + } + return { + topModel, + topCount, + topShare: reqs.length > 0 ? topCount / reqs.length : 0, + modelCount: counts.size, + total: reqs.length, + }; +} + +const CALL_DISPATCH: Record) => unknown> = { + 'length': (argNodes: ASTNode[], ctx: Record) => { + { + const v = evaluate(argNodes[0], ctx); + if (typeof v === 'string' || Array.isArray(v)) return v.length; + return 0; + } + }, + 'contains': (argNodes: ASTNode[], ctx: Record) => { + const args = () => argNodes.map(a => evaluate(a, ctx)); + + { + const [s, sub] = args(); + return typeof s === 'string' && typeof sub === 'string' && s.includes(sub); + } + }, + 'startsWith': (argNodes: ASTNode[], ctx: Record) => { + const args = () => argNodes.map(a => evaluate(a, ctx)); + + { + const [s, sub] = args(); + return typeof s === 'string' && typeof sub === 'string' && s.startsWith(sub); + } + }, + 'endsWith': (argNodes: ASTNode[], ctx: Record) => { + const args = () => argNodes.map(a => evaluate(a, ctx)); + + { + const [s, sub] = args(); + return typeof s === 'string' && typeof sub === 'string' && s.endsWith(sub); + } + }, + 'matches': (argNodes: ASTNode[], ctx: Record) => { + const args = () => argNodes.map(a => evaluate(a, ctx)); + + { + const [s, pattern] = args(); + if (typeof s !== 'string' || typeof pattern !== 'string') return false; + const m = pattern.match(/^\/(.+)\/([gimsuy]*)$/); + if (m) return testSafe(compileSafe(m[1], m[2]), s); + const pf = parseRegexFlags(pattern); + return testSafe(compileSafe(pf.pattern, pf.flags), s); + } + }, + 'lower': (argNodes: ASTNode[], ctx: Record) => { + { const v = evaluate(argNodes[0], ctx); return typeof v === 'string' ? v.toLowerCase() : ''; } + }, + 'upper': (argNodes: ASTNode[], ctx: Record) => { + { const v = evaluate(argNodes[0], ctx); return typeof v === 'string' ? v.toUpperCase() : ''; } + }, + 'trim': (argNodes: ASTNode[], ctx: Record) => { + { const v = evaluate(argNodes[0], ctx); return typeof v === 'string' ? v.trim() : ''; } + }, + 'substr': (argNodes: ASTNode[], ctx: Record) => { + const args = () => argNodes.map(a => evaluate(a, ctx)); + + { + const [s, start, len] = args(); + return typeof s === 'string' ? s.substring(toNum(start), len != null ? toNum(start) + toNum(len) : undefined) : ''; + } + }, + 'split': (argNodes: ASTNode[], ctx: Record) => { + const args = () => argNodes.map(a => evaluate(a, ctx)); + + { + const [s, sep] = args(); + return typeof s === 'string' ? s.split(asStr(sep)) : []; + } + }, + 'join': (argNodes: ASTNode[], ctx: Record) => { + const args = () => argNodes.map(a => evaluate(a, ctx)); + + { + const [arr, sep] = args(); + return Array.isArray(arr) ? arr.join(asStr(sep, ',')) : ''; + } + }, + 'truncate': (argNodes: ASTNode[], ctx: Record) => { + const args = () => argNodes.map(a => evaluate(a, ctx)); + + { + const [s, n] = args(); + if (typeof s !== 'string') return ''; + const max = toNum(n); + return s.length > max ? s.substring(0, max) + '...' : s; + } + + // Math functions + }, + 'abs': (argNodes: ASTNode[], ctx: Record) => { + return Math.abs(toNum(evaluate(argNodes[0], ctx))); + }, + 'floor': (argNodes: ASTNode[], ctx: Record) => { + return Math.floor(toNum(evaluate(argNodes[0], ctx))); + }, + 'ceil': (argNodes: ASTNode[], ctx: Record) => { + return Math.ceil(toNum(evaluate(argNodes[0], ctx))); + }, + 'round': (argNodes: ASTNode[], ctx: Record) => { + return Math.round(toNum(evaluate(argNodes[0], ctx))); + }, + 'min': (argNodes: ASTNode[], ctx: Record) => { + const args = () => argNodes.map(a => evaluate(a, ctx)); + + return Math.min(...args().map(toNum)); + }, + 'max': (argNodes: ASTNode[], ctx: Record) => { + const args = () => argNodes.map(a => evaluate(a, ctx)); + + return Math.max(...args().map(toNum)); + + // Date functions + }, + 'hour': (argNodes: ASTNode[], ctx: Record) => { + { + const ts = toNum(evaluate(argNodes[0], ctx)); + return ts > 0 ? new Date(ts).getHours() : -1; + } + }, + 'dayOfWeek': (argNodes: ASTNode[], ctx: Record) => { + { + const ts = toNum(evaluate(argNodes[0], ctx)); + return ts > 0 ? new Date(ts).getDay() : -1; + } + }, + 'dayOfMonth': (argNodes: ASTNode[], ctx: Record) => { + { + const ts = toNum(evaluate(argNodes[0], ctx)); + return ts > 0 ? new Date(ts).getDate() : -1; + } + }, + 'month': (argNodes: ASTNode[], ctx: Record) => { + { + const ts = toNum(evaluate(argNodes[0], ctx)); + return ts > 0 ? new Date(ts).getMonth() : -1; + } + }, + 'year': (argNodes: ASTNode[], ctx: Record) => { + { + const ts = toNum(evaluate(argNodes[0], ctx)); + return ts > 0 ? new Date(ts).getFullYear() : -1; + } + + // Array functions + }, + 'includes': (argNodes: ASTNode[], ctx: Record) => { + const args = () => argNodes.map(a => evaluate(a, ctx)); + + { + const [arr, val] = args(); + return Array.isArray(arr) && arr.includes(val); + } + }, + 'some': (argNodes: ASTNode[], ctx: Record) => { + { + const arr = evaluate(argNodes[0], ctx); + if (!Array.isArray(arr)) return false; + const fieldName = argNodes[1]?.type === 'identifier' ? (argNodes[1] as { name: string }).name : asStr(evaluate(argNodes[1], ctx)); + return arr.some(item => { + if (typeof item === 'object' && item) return !!(item as Record)[fieldName]; + return false; + }); + } + }, + 'every': (argNodes: ASTNode[], ctx: Record) => { + { + const arr = evaluate(argNodes[0], ctx); + if (!Array.isArray(arr)) return false; + const fieldName = argNodes[1]?.type === 'identifier' ? (argNodes[1] as { name: string }).name : asStr(evaluate(argNodes[1], ctx)); + return arr.every(item => { + if (typeof item === 'object' && item) return !!(item as Record)[fieldName]; + return false; + }); + } + }, + 'count': (argNodes: ASTNode[], ctx: Record) => { + { + const v = evaluate(argNodes[0], ctx); + return Array.isArray(v) ? v.length : 0; + } + }, + 'sum': (argNodes: ASTNode[], ctx: Record) => { + { + const arr = evaluate(argNodes[0], ctx); + if (!Array.isArray(arr)) return 0; + if (argNodes.length > 1) { + const field = argNodes[1]?.type === 'identifier' ? (argNodes[1] as { name: string }).name : asStr(evaluate(argNodes[1], ctx)); + return arr.reduce((s, item) => s + toNum((typeof item === 'object' && item ? (item as Record)[field] : 0)), 0); + } + return arr.reduce((s, item) => s + toNum(item), 0); + } + }, + 'avg': (argNodes: ASTNode[], ctx: Record) => { + { + const arr = evaluate(argNodes[0], ctx); + if (!Array.isArray(arr) || arr.length === 0) return 0; + if (argNodes.length > 1) { + const field = argNodes[1]?.type === 'identifier' ? (argNodes[1] as { name: string }).name : asStr(evaluate(argNodes[1], ctx)); + const total = arr.reduce((s, item) => s + toNum((typeof item === 'object' && item ? (item as Record)[field] : 0)), 0); + return total / arr.length; + } + return arr.reduce((s, item) => s + toNum(item), 0) / arr.length; + } + }, + 'unique': (argNodes: ASTNode[], ctx: Record) => { + { + const arr = evaluate(argNodes[0], ctx); + if (!Array.isArray(arr)) return 0; + if (argNodes.length > 1) { + const field = argNodes[1]?.type === 'identifier' ? (argNodes[1] as { name: string }).name : asStr(evaluate(argNodes[1], ctx)); + return new Set((arr as unknown[]).map(item => typeof item === 'object' && item ? (item as Record)[field] : item)).size; + } + return new Set(arr as unknown[]).size; + } + + // Object functions + }, + 'keys': (argNodes: ASTNode[], ctx: Record) => { + { + const v = evaluate(argNodes[0], ctx); + return typeof v === 'object' && v && !Array.isArray(v) ? Object.keys(v) : []; + } + }, + 'values': (argNodes: ASTNode[], ctx: Record) => { + { + const v = evaluate(argNodes[0], ctx); + return typeof v === 'object' && v && !Array.isArray(v) ? Object.values(v as Record) : []; + } + }, + 'has': (argNodes: ASTNode[], ctx: Record) => { + const args = () => argNodes.map(a => evaluate(a, ctx)); + + { + const [obj, key] = args(); + return typeof obj === 'object' && obj && !Array.isArray(obj) && asStr(key) in (obj as Record); + } + + // Utility + }, + 'coalesce': (argNodes: ASTNode[], ctx: Record) => { + { + for (const arg of argNodes) { + const v = evaluate(arg, ctx); + if (v != null) return v; + } + return null; + } + }, + 'iif': (argNodes: ASTNode[], ctx: Record) => { + { + // iif(condition, trueVal, falseVal) + const cond = toBool(evaluate(argNodes[0], ctx)); + return cond ? evaluate(argNodes[1], ctx) : (argNodes[2] ? evaluate(argNodes[2], ctx) : null); + } + }, + 'str': (argNodes: ASTNode[], ctx: Record) => { + { + const v = evaluate(argNodes[0], ctx); + return asStr(v); + } + }, + 'num': (argNodes: ASTNode[], ctx: Record) => { + { + const v = evaluate(argNodes[0], ctx); + return toNum(v); + } + + // ── Extended functions for rule pipeline ── + + /** matchesAny(text, patternArray) – true if text matches ANY regex in the array */ + }, + 'matchesAny': (argNodes: ASTNode[], ctx: Record) => { + const args = () => argNodes.map(a => evaluate(a, ctx)); + + { + const [text, patterns] = args(); + if (typeof text !== 'string' || !Array.isArray(patterns)) return false; + return patterns.some(p => { + const pf = parseRegexFlags(asStr(p)); + return testSafe(compileSafe(pf.pattern, pf.flags), text); + }); + } + + /** capsLetterRatio(text) – ratio of uppercase letters to total letters */ + }, + 'capsLetterRatio': (argNodes: ASTNode[], ctx: Record) => { + { + const text = asStr(evaluate(argNodes[0], ctx)); + let letters = 0, upper = 0; + for (let i = 0, len = Math.min(text.length, 2000); i < len; i++) { + const c = text.charCodeAt(i); + if (c >= 65 && c <= 90) { letters++; upper++; } + else if (c >= 97 && c <= 122) { letters++; } + } + return letters > 0 ? upper / letters : 0; + } + + /** capsWordRatio(text, minWordLen?) – ratio of ALL-CAPS words (default minWordLen=3) */ + }, + 'capsWordRatio': (argNodes: ASTNode[], ctx: Record) => { + { + const text = asStr(evaluate(argNodes[0], ctx)); + const minLen = argNodes.length > 1 ? toNum(evaluate(argNodes[1], ctx)) : 3; + let wordLen = 0, wordCount = 0, capsWordCount = 0; + let allUpper = true, hasLetter = false; + const scanLen = Math.min(text.length, 2000); + for (let i = 0; i <= scanLen; i++) { + const c = i < scanLen ? text.charCodeAt(i) : 32; + if (c === 32 || c === 9 || c === 10 || c === 13) { + if (wordLen >= minLen) { + wordCount++; + if (allUpper && hasLetter) capsWordCount++; + } + wordLen = 0; allUpper = true; hasLetter = false; + } else { + wordLen++; + if (c >= 65 && c <= 90) hasLetter = true; + else if (c >= 97 && c <= 122) allUpper = false; + } + } + return wordCount > 0 ? capsWordCount / wordCount : 0; + } + + /** sumLoc(codeBlocks) – sum .loc from an array of {language, loc} blocks */ + }, + 'sumLoc': (argNodes: ASTNode[], ctx: Record) => { + { + const arr = evaluate(argNodes[0], ctx); + if (!Array.isArray(arr)) return 0; + return arr.reduce((s, b) => s + toNum((b as Record)?.loc), 0); + } + + /** sumLocFor(codeBlocks, langArray) – sum .loc where .language is in the array */ + }, + 'sumLocFor': (argNodes: ASTNode[], ctx: Record) => { + const args = () => argNodes.map(a => evaluate(a, ctx)); + + { + const [arr, langs] = args(); + if (!Array.isArray(arr) || !Array.isArray(langs)) return 0; + const langSet = new Set(langs.map((l: unknown) => asStr(l).toLowerCase())); + return (arr as Record[]) + .filter(b => langSet.has(asStr(b.language).toLowerCase())) + .reduce((s, b) => s + toNum(b.loc), 0); + } + + /** modelTier(modelId) – returns model cost multiplier (0 = free/unknown, ≥1 = premium) */ + }, + 'modelTier': (argNodes: ASTNode[], ctx: Record) => { + { + const id = asStr(evaluate(argNodes[0], ctx)); + return modelTierLookup(id); + } + + /** classifyWork(text) – classify message into work type string */ + }, + 'classifyWork': (argNodes: ASTNode[], ctx: Record) => { + { + const text = asStr(evaluate(argNodes[0], ctx)); + return classifyWorkText(text); + } + + // ── Array access functions ── + + /** first(arr) – first element of array */ + }, + 'first': (argNodes: ASTNode[], ctx: Record) => { + { + const arr = evaluate(argNodes[0], ctx); + return Array.isArray(arr) && arr.length > 0 ? (arr as unknown[])[0] : null; + } + + /** at(arr, idx) – element at index */ + }, + 'at': (argNodes: ASTNode[], ctx: Record) => { + const args = () => argNodes.map(a => evaluate(a, ctx)); + + { + const [arr, idx] = args(); + return Array.isArray(arr) ? (arr as unknown[])[toNum(idx)] ?? null : null; + } + + // ── String analysis functions ── + + /** substring(str, start, end) – extract portion of string */ + }, + 'substring': (argNodes: ASTNode[], ctx: Record) => { + const args = () => argNodes.map(a => evaluate(a, ctx)); + + { + const [s, start, end] = args(); + return typeof s === 'string' ? s.substring(toNum(start), end != null ? toNum(end) : undefined) : ''; + } + + /** lineCount(str) – count non-empty lines */ + }, + 'lineCount': (argNodes: ASTNode[], ctx: Record) => { + { + const s = asStr(evaluate(argNodes[0], ctx)); + return s.split('\n').filter(l => l.trim().length > 0).length; + } + + /** isStructured(str_or_reqObj) – true if text has bullets, numbered lists, headings, or spec keywords */ + }, + 'isStructured': (argNodes: ASTNode[], ctx: Record) => { + { + const raw = evaluate(argNodes[0], ctx); + const s = (typeof raw === 'object' && raw && !Array.isArray(raw)) + ? asStr((raw as Record).messageText) + : asStr(raw); + return /^[-*]\s/m.test(s) || // bullets + /^\d+[.)]\s/m.test(s) || // numbered lists + /^#+\s/m.test(s) || // markdown headings + /\b(requirements?|spec|acceptance criteria|user stor(y|ies)|given|when|then|should|must)\b/i.test(s) || + s.split('\n').filter(l => l.trim().length > 0).length >= 4; + } + + /** hasProfanity(str) – detect profanity in text (strips code blocks first) */ + }, + 'hasProfanity': (argNodes: ASTNode[], ctx: Record) => { + { + const raw = asStr(evaluate(argNodes[0], ctx)); + return detectProfanity(raw); + } + + /** normalizeModel(id) – normalize model ID string */ + }, + 'normalizeModel': (argNodes: ASTNode[], ctx: Record) => { + { + const raw = asStr(evaluate(argNodes[0], ctx)); + return normalizeModelId(raw); + } + + // ── Generic array/regex utility functions ── + + /** anyMatch(strArr, pattern) – 1 if any string in array matches regex */ + }, + 'anyMatch': (argNodes: ASTNode[], ctx: Record) => { + { + const arr = evaluate(argNodes[0], ctx); + const pattern = asStr(evaluate(argNodes[1], ctx)); + if (!Array.isArray(arr)) return 0; + const pf = parseRegexFlags(pattern); + const re = compileSafe(pf.pattern, pf.flags); + if (!re) return 0; + return arr.some(item => testSafe(re, asStr(item))) ? 1 : 0; + } + + /** flatSumField(arr, arrayField, sumField) – sum field across nested arrays */ + }, + 'flatSumField': (argNodes: ASTNode[], ctx: Record) => { + { + const arr = evaluate(argNodes[0], ctx); + if (!Array.isArray(arr)) return 0; + const arrayField = argNodes[1]?.type === 'identifier' + ? (argNodes[1] as { name: string }).name : asStr(evaluate(argNodes[1], ctx)); + const sumFieldName = argNodes[2]?.type === 'identifier' + ? (argNodes[2] as { name: string }).name : asStr(evaluate(argNodes[2], ctx)); + let total = 0; + for (const item of arr) { + const sub = resolveField(item as Record, arrayField); + if (Array.isArray(sub)) { + for (const elem of sub) { + total += toNum(resolveField(elem as Record, sumFieldName)); + } + } + } + return total; + } + + /** slice(arr, start, end?) – returns arr.slice(start, end) */ + }, + 'slice': (argNodes: ASTNode[], ctx: Record) => { + { + const arr = evaluate(argNodes[0], ctx); + if (!Array.isArray(arr)) return []; + const start = toNum(evaluate(argNodes[1], ctx)); + const end = argNodes.length > 2 ? toNum(evaluate(argNodes[2], ctx)) : undefined; + return (arr as unknown[]).slice(start, end); + } + + /** anyWhere(arr, field, op, value) – 1 if any element matches, supports "matches" op */ + }, + 'anyWhere': (argNodes: ASTNode[], ctx: Record) => { + { + const arr = evaluate(argNodes[0], ctx); + if (!Array.isArray(arr)) return 0; + const fieldName = asStr(evaluate(argNodes[1], ctx)); + const op = argNodes.length >= 4 ? asStr(evaluate(argNodes[2], ctx)) : '=='; + const value = argNodes.length >= 4 ? evaluate(argNodes[3], ctx) : evaluate(argNodes[2], ctx); + return (arr as unknown[]).some(item => { + const row: Record = typeof item === 'object' && item ? item as Record : { '.': item }; + const fv: unknown = fieldName === '.' ? item : resolveField(row, fieldName); + switch (op) { + case '==': return looseEquals(fv, value); + case '!=': return !looseEquals(fv, value); + case '>': return toNum(fv) > toNum(value); + case '>=': return toNum(fv) >= toNum(value); + case '<': return toNum(fv) < toNum(value); + case '<=': return toNum(fv) <= toNum(value); + case 'matches': { + const pf = parseRegexFlags(asStr(value)); + return testSafe(compileSafe(pf.pattern, pf.flags), asStr(fv)); + } + default: return looseEquals(fv, value); + } + }) ? 1 : 0; + } + }, + 'sumAiLoc': (argNodes: ASTNode[], ctx: Record) => { + { + const reqs = evaluate(argNodes[0], ctx); + if (!Array.isArray(reqs)) return 0; + let total = 0; + for (const r of reqs) { + const blocks = (r as Record)?.aiCode; + if (Array.isArray(blocks)) { + for (const b of blocks) total += toNum((b as Record)?.loc); + } + } + return total; + } + }, + 'workTypeCount': (argNodes: ASTNode[], ctx: Record) => { + { + const reqs = evaluate(argNodes[0], ctx); + if (!Array.isArray(reqs)) return 0; + const types = new Set(); + for (const r of reqs) { + const msg = asStr((r as Record)?.messageText); + const wt = (r as Record)?.workType; + types.add(typeof wt === 'string' && wt ? wt : classifyWorkText(msg)); + } + return types.size; + } + }, + 'hasReviewFollowup': (argNodes: ASTNode[], ctx: Record) => { + { + const reqs = evaluate(argNodes[0], ctx); + if (!Array.isArray(reqs) || reqs.length < 2) return false; + const REVIEW_RE = /\b(change|fix|modify|update|refactor|wrong|instead|actually|revert|redo|try again)\b/i; + for (let i = 1; i < reqs.length; i++) { + const r = reqs[i] as Record; + const msg = asStr(r?.messageText); + const edits = r?.editedFiles; + if (REVIEW_RE.test(msg) || (Array.isArray(edits) && edits.length > 0)) return true; + } + return false; + } + }, + 'groupTopKey': (argNodes: ASTNode[], ctx: Record) => { + { + const arr = evaluate(argNodes[0], ctx); + if (!Array.isArray(arr) || arr.length === 0) return ''; + const fieldName = argNodes[1]?.type === 'identifier' ? (argNodes[1] as { name: string }).name : asStr(evaluate(argNodes[1], ctx)); + return groupTop(arr as unknown[], fieldName).key; + } + }, + 'groupTopCount': (argNodes: ASTNode[], ctx: Record) => { + { + const arr = evaluate(argNodes[0], ctx); + if (!Array.isArray(arr) || arr.length === 0) return 0; + const fieldName = argNodes[1]?.type === 'identifier' ? (argNodes[1] as { name: string }).name : asStr(evaluate(argNodes[1], ctx)); + return groupTop(arr as unknown[], fieldName).count; + } + }, + 'groupTopShare': (argNodes: ASTNode[], ctx: Record) => { + { + const arr = evaluate(argNodes[0], ctx); + if (!Array.isArray(arr) || arr.length === 0) return 0; + const fieldName = argNodes[1]?.type === 'identifier' ? (argNodes[1] as { name: string }).name : asStr(evaluate(argNodes[1], ctx)); + return groupTop(arr as unknown[], fieldName).share; + } + }, + 'countWhere': (argNodes: ASTNode[], ctx: Record) => { + { + const arr = evaluate(argNodes[0], ctx); + if (!Array.isArray(arr)) return 0; + const fieldName = asStr(evaluate(argNodes[1], ctx)); + const op = asStr(evaluate(argNodes[2], ctx)); + const value = evaluate(argNodes[3], ctx); + return arr.filter(item => { + const fv = resolveField(item as Record, fieldName); + switch (op) { + case '>': return toNum(fv) > toNum(value); + case '>=': return toNum(fv) >= toNum(value); + case '<': return toNum(fv) < toNum(value); + case '<=': return toNum(fv) <= toNum(value); + case '==': return looseEquals(fv, value); + case '!=': return !looseEquals(fv, value); + case 'matches': { + const pf = parseRegexFlags(asStr(value)); + return testSafe(compileSafe(pf.pattern, pf.flags), asStr(fv)); + } + default: return toBool(fv); + } + }).length; + } + }, + 'sumField': (argNodes: ASTNode[], ctx: Record) => { + { + const arr = evaluate(argNodes[0], ctx); + if (!Array.isArray(arr)) return 0; + const fieldName = argNodes[1]?.type === 'identifier' ? (argNodes[1] as { name: string }).name : asStr(evaluate(argNodes[1], ctx)); + return arr.reduce((s, item) => s + toNum(resolveField(item as Record, fieldName)), 0); + } + }, + 'avgField': (argNodes: ASTNode[], ctx: Record) => { + { + const arr = evaluate(argNodes[0], ctx); + if (!Array.isArray(arr) || arr.length === 0) return 0; + const fieldName = argNodes[1]?.type === 'identifier' ? (argNodes[1] as { name: string }).name : asStr(evaluate(argNodes[1], ctx)); + const total = arr.reduce((s, item) => s + toNum(resolveField(item as Record, fieldName)), 0); + return total / arr.length; + } + }, + 'flatCount': (argNodes: ASTNode[], ctx: Record) => { + { + const arr = evaluate(argNodes[0], ctx); + if (!Array.isArray(arr)) return 0; + const fieldName = argNodes[1]?.type === 'identifier' ? (argNodes[1] as { name: string }).name : asStr(evaluate(argNodes[1], ctx)); + return arr.reduce((s, item) => { + const sub = resolveField(item as Record, fieldName); + return s + (Array.isArray(sub) ? sub.length : 0); + }, 0); + } + }, + 'flatUnique': (argNodes: ASTNode[], ctx: Record) => { + { + const arr = evaluate(argNodes[0], ctx); + if (!Array.isArray(arr)) return 0; + const fieldName = argNodes[1]?.type === 'identifier' ? (argNodes[1] as { name: string }).name : asStr(evaluate(argNodes[1], ctx)); + const seen = new Set(); + for (const item of arr) { + const sub = resolveField(item as Record, fieldName); + if (Array.isArray(sub)) { + for (const v of sub) { + if (v !== null && v !== undefined) seen.add(asStr(v)); + } + } + } + return seen.size; + } + }, + 'flatSomeWhere': (argNodes: ASTNode[], ctx: Record) => { + { + const arr = evaluate(argNodes[0], ctx); + if (!Array.isArray(arr)) return 0; + const fieldName = argNodes[1]?.type === 'identifier' ? (argNodes[1] as { name: string }).name : asStr(evaluate(argNodes[1], ctx)); + const subField = asStr(evaluate(argNodes[2], ctx)); + const target = evaluate(argNodes[3], ctx); + const op = argNodes.length >= 5 ? asStr(evaluate(argNodes[4], ctx)) : '=='; + const checkElem = (elem: unknown): boolean => { + const fv = subField === '.' ? elem : resolveField(elem as Record, subField); + switch (op) { + case '==': return looseEquals(fv, target); + case '!=': return !looseEquals(fv, target); + case 'startsWith': return asStr(fv).startsWith(asStr(target)); + case 'contains': return asStr(fv).includes(asStr(target)); + case 'matches': { + const pf = parseRegexFlags(asStr(target)); + return testSafe(compileSafe(pf.pattern, pf.flags), asStr(fv)); + } + default: return looseEquals(fv, target); + } + }; + return arr.some(item => { + const sub = resolveField(item as Record, fieldName); + if (!Array.isArray(sub)) return false; + return sub.some(checkElem); + }) ? 1 : 0; + } + }, + 'adjacentPairCount': (argNodes: ASTNode[], ctx: Record) => { + { + const sessions = evaluate(argNodes[0], ctx); + const minLoc = toNum(evaluate(argNodes[1], ctx)); + const maxGap = toNum(evaluate(argNodes[2], ctx)); + if (!Array.isArray(sessions)) return { count: 0, avgLoc: 0, avgGap: 0 }; + return computeSpeedAcceptPairs(sessions as Record[], minLoc, maxGap); + } + }, + 'flowScoreStats': (argNodes: ASTNode[], ctx: Record) => { + { + const sessions = evaluate(argNodes[0], ctx); + const minTimed = argNodes.length > 1 ? toNum(evaluate(argNodes[1], ctx)) : 3; + const rapidMs = argNodes.length > 2 ? toNum(evaluate(argNodes[2], ctx)) : 30000; + if (!Array.isArray(sessions)) return { fragmentedDays: 0, totalDays: 0, avgScore: 100 }; + return computeFlowScoreStats(sessions as Record[], minTimed, rapidMs); + } + }, + 'langExplorationWeeks': (argNodes: ASTNode[], ctx: Record) => { + { + const reqs = evaluate(argNodes[0], ctx); + if (!Array.isArray(reqs)) return { weeksSinceNew: 0, totalLangs: 0, recentNew: 0 }; + return computeLangExploration(reqs as Record[]); + } + }, + 'mdRatioByWorkspace': (argNodes: ASTNode[], ctx: Record) => { + { + const sessions = evaluate(argNodes[0], ctx); + const minLoc = argNodes.length > 1 ? toNum(evaluate(argNodes[1], ctx)) : 100; + const docLangs = argNodes.length > 2 ? evaluate(argNodes[2], ctx) : ['markdown', 'md']; + if (!Array.isArray(sessions)) return { lowCount: 0, totalWorkspaces: 0, overallRatio: 0 }; + return computeMdRatio( + sessions as Record[], + minLoc, + Array.isArray(docLangs) ? docLangs.map(String) : ['markdown', 'md'], + ); + } + }, + 'devcontainerStats': (argNodes: ASTNode[], ctx: Record) => { + { + const sessions = evaluate(argNodes[0], ctx); + const reqs = evaluate(argNodes[1], ctx); + if (!Array.isArray(sessions) || !Array.isArray(reqs)) return { terminalReqs: 0, vscodeReqs: 0, sandboxedTerminalReqs: 0, totalTerminalReqs: 0, terminalRate: 0 }; + return computeDevcontainerStats(sessions as Record[], reqs as Record[]); + } + }, + 'contextGapCount': (argNodes: ASTNode[], ctx: Record) => { + { + const reqs = evaluate(argNodes[0], ctx); + if (!Array.isArray(reqs)) return { gapCount: 0, gaps: [], reqCount: 0 }; + return computeContextGaps(reqs as Record[]); + } + }, + 'yoloStats': (argNodes: ASTNode[], ctx: Record) => { + { + const reqs = evaluate(argNodes[0], ctx); + if (!Array.isArray(reqs)) return { autoApproved: 0, totalConfirmations: 0, ratio: 0 }; + return computeYoloStats(reqs as Record[]); + } + }, + 'reasoningEffortStats': (argNodes: ASTNode[], ctx: Record) => { + { + const reqs = evaluate(argNodes[0], ctx); + const premiumLevel = argNodes.length > 1 ? asStr(evaluate(argNodes[1], ctx), 'high').toLowerCase() : 'high'; + if (!Array.isArray(reqs)) return { premiumCount: 0, totalKnown: 0, total: 0, ratio: 0, topTier: '' }; + return computeReasoningEffortStats(reqs as Record[], premiumLevel); + } + }, + 'instructionBloatStats': (argNodes: ASTNode[], ctx: Record) => { + { + const sessions = evaluate(argNodes[0], ctx); + const maxBytes = argNodes.length > 1 ? toNum(evaluate(argNodes[1], ctx)) : 4000; + if (!Array.isArray(sessions)) return { bloatedSessions: 0, withInstructionsCount: 0, totalSessions: 0, maxBytes: 0, p95Bytes: 0 }; + return computeInstructionBloatStats(sessions as Record[], maxBytes); + } + }, + 'excessFileContextStats': (argNodes: ASTNode[], ctx: Record) => { + { + const reqs = evaluate(argNodes[0], ctx); + const minFiles = argNodes.length > 1 ? toNum(evaluate(argNodes[1], ctx)) : 30; + if (!Array.isArray(reqs)) return { outlierCount: 0, totalReqs: 0, ratio: 0, p95Files: 0, maxFiles: 0 }; + return computeExcessFileContextStats(reqs as Record[], minFiles); + } + }, + 'hasSkillByPattern': (argNodes: ASTNode[], ctx: Record) => { + { + const reqs = evaluate(argNodes[0], ctx); + const pattern = asStr(evaluate(argNodes[1], ctx)); + if (!Array.isArray(reqs) || !pattern) return 0; + const pf = parseRegexFlags(pattern); + const re = compileSafe(pf.pattern, pf.flags); + if (!re) return 0; + for (const r of reqs) { + const skills = (r as Record).skillsUsed; + if (Array.isArray(skills)) { + for (const s of skills) { + if (testSafe(re, asStr(s))) return 1; + } + } + } + return 0; + } + }, + 'duplicateGroups': (argNodes: ASTNode[], ctx: Record) => { + { + const reqs = evaluate(argNodes[0], ctx); + const minKeyLen = argNodes.length > 1 ? toNum(evaluate(argNodes[1], ctx)) : 10; + const minCount = argNodes.length > 2 ? toNum(evaluate(argNodes[2], ctx)) : 3; + if (!Array.isArray(reqs)) return { totalDupes: 0, distinctCount: 0, topKey: '', topCount: 0 }; + return computeDuplicateGroups(reqs as Record[], minKeyLen, minCount); + } + }, + 'profanityMatches': (argNodes: ASTNode[], ctx: Record) => { + { + const reqs = evaluate(argNodes[0], ctx); + if (!Array.isArray(reqs)) return { count: 0, total: 0, flaggedWords: [] }; + return computeProfanityMatches(reqs as Record[]); + } + }, + 'hasConstraint': (argNodes: ASTNode[], ctx: Record) => { + { + const text = asStr(evaluate(argNodes[0], ctx)); + const sample = text.length > 500 ? text.substring(0, 500) : text; + return /\b(do not|don't|dont|must not|mustn't|never|without|avoid|only|strictly|limit to|at most|at least|no more than|require|restrict|exclude|ensure|must|shall|should not|shouldn't)\b/i.test(sample); + } + }, + 'someWhere': (argNodes: ASTNode[], ctx: Record) => { + { + const arr = evaluate(argNodes[0], ctx); + if (!Array.isArray(arr)) return false; + const fieldName = asStr(evaluate(argNodes[1], ctx)); + const op = argNodes.length >= 4 ? asStr(evaluate(argNodes[2], ctx)) : '=='; + const value = argNodes.length >= 4 ? evaluate(argNodes[3], ctx) : evaluate(argNodes[2], ctx); + return arr.some(item => { + if (typeof item !== 'object' || !item) return false; + const fv = resolveField(item as Record, fieldName); + switch (op) { + case '==': return looseEquals(fv, value); + case '!=': return !looseEquals(fv, value); + case '>': return toNum(fv) > toNum(value); + case '>=': return toNum(fv) >= toNum(value); + case '<': return toNum(fv) < toNum(value); + case '<=': return toNum(fv) <= toNum(value); + case 'matches': { + const pf = parseRegexFlags(asStr(value)); + return testSafe(compileSafe(pf.pattern, pf.flags), asStr(fv)); + } + default: return looseEquals(fv, value); + } + }) ? 1 : 0; + } + }, + 'isSpecDriven': (argNodes: ASTNode[], ctx: Record) => { + { + const arr = evaluate(argNodes[0], ctx); + if (!Array.isArray(arr) || arr.length === 0) return 0; + const first = arr[0] as Record; + + const refs = first.referencedFiles; + if (Array.isArray(refs) && refs.some(f => /\.(md|txt|spec|prd|design|plan|rfc|adoc)$/i.test(asStr(f)))) return 1; + + const msg = asStr(first.messageText); + if (/\b(spec|requirements?|acceptance criteria|design doc|PRD|RFC|plan file|constraint|must|should|ensure)\b/i.test(msg)) return 1; + if (/^[-*]\s/m.test(msg) && msg.split('\n').filter(l => l.trim()).length >= 3) return 1; + if (/^\d+[.)]\s/m.test(msg) && msg.split('\n').filter(l => l.trim()).length >= 3) return 1; + if (/^#+\s/m.test(msg)) return 1; + + const mode = asStr(first.agentMode); + const slash = asStr(first.slashCommand); + if (mode.includes('plan') || slash === 'plan') return 1; + + return 0; + } + }, + 'hasPlanUsage': (argNodes: ASTNode[], ctx: Record) => { + { + const arr = evaluate(argNodes[0], ctx); + if (!Array.isArray(arr)) return 0; + return (arr as Record[]).some(r => { + const mode = asStr(r.agentMode); + const slash = asStr(r.slashCommand); + if (mode.includes('plan') || slash === 'plan') return true; + const msg = asStr(r.messageText).toLowerCase(); + return msg.startsWith('plan') && toNum(r.messageLength) > 15; + }) ? 1 : 0; + } + }, + 'autoApproveStats': (argNodes: ASTNode[], ctx: Record) => { + { + const reqs = evaluate(argNodes[0], ctx); + if (!Array.isArray(reqs)) return { terminalAutoApproved: 0, autoApprovedTotal: 0, withConfirmations: 0 }; + return computeAutoApproveStats(reqs as Record[]); + } + }, + 'groupTopBySum': (argNodes: ASTNode[], ctx: Record) => { + { + const arr = evaluate(argNodes[0], ctx); + if (!Array.isArray(arr) || arr.length === 0) return { key: '', sum: 0, share: 0 }; + const groupField = asStr(evaluate(argNodes[1], ctx)); + const sumField = asStr(evaluate(argNodes[2], ctx)); + return groupTopBySum(arr as unknown[], groupField, sumField); + } + }, + 'uniqueCount': (argNodes: ASTNode[], ctx: Record) => { + { + const arr = evaluate(argNodes[0], ctx); + if (!Array.isArray(arr)) return 0; + const fieldName = asStr(evaluate(argNodes[1], ctx)); + const vals = new Set(arr.map(item => + typeof item === 'object' && item ? asStr(resolveField(item as Record, fieldName)) : '' + )); + return vals.size; + } + }, + 'modelStats': (argNodes: ASTNode[], ctx: Record) => { + { + const reqs = evaluate(argNodes[0], ctx); + if (!Array.isArray(reqs)) return { topModel: '', topCount: 0, topShare: 0, modelCount: 0, total: 0 }; + return computeModelStats(reqs as Record[]); + } + }, + 'hasPlanning': (argNodes: ASTNode[], ctx: Record) => { + { + const arr = evaluate(argNodes[0], ctx); + if (!Array.isArray(arr)) return 0; + return (arr as Record[]).some(r => { + const mode = asStr(r.agentMode); + const slash = asStr(r.slashCommand); + if (mode.includes('plan') || slash === 'plan') return true; + const msg = asStr(r.messageText).substring(0, 300); + return /\b(plan|architect|design|outline|approach|strategy|scope|breakdown|roadmap|RFC|spec|proposal)\b/i.test(msg); + }) ? 1 : 0; + } + }, +}; diff --git a/packages/ai-coach-analytics/src/dsl/lexer.ts b/packages/ai-coach-analytics/src/dsl/lexer.ts new file mode 100644 index 0000000..eeaeebc --- /dev/null +++ b/packages/ai-coach-analytics/src/dsl/lexer.ts @@ -0,0 +1,218 @@ +/*--------------------------------------------------------------------------------------------- + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the MIT License. See LICENSE in the project root for license information. + *--------------------------------------------------------------------------------------------*/ + +/** + * Lexer for the metric expression DSL. + * Converts a source string into a list of tokens. + */ + +import { Token, TokenType } from './types'; + +const KEYWORDS: Record = { + and: TokenType.AND, + or: TokenType.OR, + not: TokenType.NOT, + true: TokenType.BOOLEAN, + false: TokenType.BOOLEAN, + AND: TokenType.AND, + OR: TokenType.OR, + NOT: TokenType.NOT, + TRUE: TokenType.BOOLEAN, + FALSE: TokenType.BOOLEAN, +}; + +const TWO_CHAR_TOKENS: Record = { + '<=': TokenType.LTE, + '>=': TokenType.GTE, + '==': TokenType.EQ, + '!=': TokenType.NEQ, +}; + +const SINGLE_CHAR_TOKENS: Record = { + '<': TokenType.LT, + '>': TokenType.GT, + '(': TokenType.LPAREN, + ')': TokenType.RPAREN, + '[': TokenType.LBRACKET, + ']': TokenType.RBRACKET, + '.': TokenType.DOT, + ',': TokenType.COMMA, + '|': TokenType.PIPE, + ':': TokenType.COLON, + '+': TokenType.PLUS, + '*': TokenType.STAR, +}; + +const REGEX_FLAG_RE = /[gimsuy]/; + +export class LexerError extends Error { + constructor(message: string, public pos: number) { + super(`Lexer error at position ${pos}: ${message}`); + } +} + +interface ScanResult { + token: Token; + nextIndex: number; +} + +export function lex(input: string): Token[] { + const tokens: Token[] = []; + let index = 0; + + while (index < input.length) { + index = skipWhitespace(input, index); + if (index >= input.length) break; + + const prevToken = tokens[tokens.length - 1]; + const token = + readTwoCharToken(input, index) ?? + readSingleCharToken(input, index) ?? + readSlashToken(input, index, prevToken) ?? + readMinusToken(input, index, prevToken) ?? + readNumberToken(input, index) ?? + readQuotedStringToken(input, index) ?? + readIdentifierToken(input, index); + + if (!token) { + throw new LexerError(`Unexpected character '${input[index]}'`, index); + } + + tokens.push(token.token); + index = token.nextIndex; + } + + tokens.push({ type: TokenType.EOF, value: '', pos: input.length }); + return tokens; +} + +function skipWhitespace(input: string, start: number): number { + let index = start; + while (index < input.length && isWhitespace(input[index])) index++; + return index; +} + +function isWhitespace(char: string): boolean { + return char === ' ' || char === '\t' || char === '\r' || char === '\n'; +} + +function createToken(type: TokenType, value: string, pos: number, nextIndex: number): ScanResult { + return { token: { type, value, pos }, nextIndex }; +} + +function readTwoCharToken(input: string, index: number): ScanResult | null { + const type = TWO_CHAR_TOKENS[input.slice(index, index + 2)]; + return type === undefined ? null : createToken(type, input.slice(index, index + 2), index, index + 2); +} + +function readSingleCharToken(input: string, index: number): ScanResult | null { + const char = input[index]; + const type = SINGLE_CHAR_TOKENS[char]; + return type === undefined ? null : createToken(type, char, index, index + 1); +} + +function isDivisionContext(token: Token | undefined): boolean { + return token?.type === TokenType.NUMBER || + token?.type === TokenType.IDENTIFIER || + token?.type === TokenType.RPAREN || + token?.type === TokenType.RBRACKET; +} + +function readSlashToken(input: string, index: number, prevToken: Token | undefined): ScanResult | null { + if (input[index] !== '/') return null; + if (isDivisionContext(prevToken)) { + return createToken(TokenType.SLASH, '/', index, index + 1); + } + return readRegexToken(input, index); +} + +function isUnaryMinusContext(token: Token | undefined): boolean { + return token === undefined || + token.type === TokenType.LPAREN || + token.type === TokenType.COMMA || + token.type === TokenType.LT || + token.type === TokenType.GT || + token.type === TokenType.LTE || + token.type === TokenType.GTE || + token.type === TokenType.EQ || + token.type === TokenType.NEQ || + token.type === TokenType.AND || + token.type === TokenType.OR || + token.type === TokenType.NOT || + token.type === TokenType.PLUS || + token.type === TokenType.MINUS || + token.type === TokenType.STAR || + token.type === TokenType.SLASH; +} + +function readMinusToken(input: string, index: number, prevToken: Token | undefined): ScanResult | null { + if (input[index] !== '-') return null; + if (isUnaryMinusContext(prevToken) && isDigit(input[index + 1] ?? '')) { + return readNumberToken(input, index, true); + } + return createToken(TokenType.MINUS, '-', index, index + 1); +} + +function readNumberToken(input: string, index: number, allowLeadingMinus = false): ScanResult | null { + const firstChar = input[index]; + if (!isDigit(firstChar) && !(allowLeadingMinus && firstChar === '-')) return null; + let nextIndex = index + 1; + while (nextIndex < input.length && (isDigit(input[nextIndex]) || input[nextIndex] === '.')) { + nextIndex++; + } + return createToken(TokenType.NUMBER, input.slice(index, nextIndex), index, nextIndex); +} + +function readQuotedStringToken(input: string, index: number): ScanResult | null { + const quote = input[index]; + if (quote !== '"' && quote !== "'") return null; + let nextIndex = index + 1; + let value = ''; + while (nextIndex < input.length && input[nextIndex] !== quote) { + if (input[nextIndex] === '\\' && nextIndex + 1 < input.length) nextIndex++; + value += input[nextIndex]; + nextIndex++; + } + if (nextIndex < input.length) nextIndex++; + return createToken(TokenType.STRING, value, index, nextIndex); +} + +function readRegexToken(input: string, index: number): ScanResult { + let nextIndex = index + 1; + let pattern = ''; + while (nextIndex < input.length && input[nextIndex] !== '/') { + if (input[nextIndex] === '\\' && nextIndex + 1 < input.length) { + pattern += input[nextIndex]; + nextIndex++; + } + pattern += input[nextIndex]; + nextIndex++; + } + if (nextIndex < input.length) nextIndex++; + const flagsStart = nextIndex; + while (nextIndex < input.length && REGEX_FLAG_RE.test(input[nextIndex])) nextIndex++; + const flags = input.slice(flagsStart, nextIndex); + return createToken(TokenType.STRING, `/${pattern}/${flags}`, index, nextIndex); +} + +function readIdentifierToken(input: string, index: number): ScanResult | null { + if (!isIdentStart(input[index])) return null; + let nextIndex = index + 1; + while (nextIndex < input.length && isIdentPart(input[nextIndex])) nextIndex++; + const value = input.slice(index, nextIndex); + return createToken(KEYWORDS[value] ?? TokenType.IDENTIFIER, value, index, nextIndex); +} + +function isDigit(c: string): boolean { + return c >= '0' && c <= '9'; +} + +function isIdentStart(c: string): boolean { + return (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || c === '_'; +} + +function isIdentPart(c: string): boolean { + return isIdentStart(c) || isDigit(c); +} diff --git a/packages/ai-coach-analytics/src/dsl/parser.ts b/packages/ai-coach-analytics/src/dsl/parser.ts new file mode 100644 index 0000000..21391e3 --- /dev/null +++ b/packages/ai-coach-analytics/src/dsl/parser.ts @@ -0,0 +1,252 @@ +/*--------------------------------------------------------------------------------------------- + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the MIT License. See LICENSE in the project root for license information. + *--------------------------------------------------------------------------------------------*/ + +/** + * Recursive-descent parser for the metric expression DSL. + * Turns a token stream into an AST. + * + * Grammar (simplified): + * expr := or_expr + * or_expr := and_expr ('OR' and_expr)* + * and_expr := not_expr ('AND' not_expr)* + * not_expr := 'NOT' not_expr | comparison + * comparison := addition (('<'|'>'|'<='|'>='|'=='|'!=') addition)? + * addition := unary (('+' | '-') unary)* + * unary := pipe_expr + * pipe_expr := primary ('|' IDENT (':' primary)?)* + * primary := NUMBER | STRING | BOOLEAN | call_or_field | '(' expr ')' | '[' list ']' + * call_or_field := IDENT ('(' args ')' | ('.' IDENT)*) + * args := expr (',' expr)* + */ + +import { Token, TokenType, ASTNode } from './types'; + +export class ParseError extends Error { + constructor(message: string, public pos: number) { + super(`Parse error at position ${pos}: ${message}`); + } +} + +const MAX_PARSE_DEPTH = 64; + +export function parse(tokens: Token[]): ASTNode { + let pos = 0; + let depth = 0; + + function peek(): Token { return tokens[pos]; } + function advance(): Token { return tokens[pos++]; } + function expect(type: TokenType): Token { + const t = peek(); + if (t.type !== type) { + throw new ParseError(`Expected ${TokenType[type]}, got ${TokenType[t.type]} '${t.value}'`, t.pos); + } + return advance(); + } + function enter(): void { + depth++; + if (depth > MAX_PARSE_DEPTH) { + throw new ParseError(`Expression too deeply nested (max depth ${MAX_PARSE_DEPTH})`, peek().pos); + } + } + function exit(): void { depth--; } + + function parseExpr(): ASTNode { + return parseOr(); + } + + function parseOr(): ASTNode { + enter(); + try { + let left = parseAnd(); + while (peek().type === TokenType.OR) { + advance(); + const right = parseAnd(); + left = { type: 'binary', op: 'OR', left, right }; + } + return left; + } finally { exit(); } + } + + function parseAnd(): ASTNode { + enter(); + try { + let left = parseNot(); + while (peek().type === TokenType.AND) { + advance(); + const right = parseNot(); + left = { type: 'binary', op: 'AND', left, right }; + } + return left; + } finally { exit(); } + } + + function parseNot(): ASTNode { + enter(); + try { + if (peek().type === TokenType.NOT) { + advance(); + const operand = parseNot(); + return { type: 'unary', op: 'NOT', operand }; + } + return parseComparison(); + } finally { exit(); } + } + + function parseComparison(): ASTNode { + enter(); + try { + const left = parseAddition(); + const t = peek(); + if (t.type === TokenType.LT || t.type === TokenType.GT || + t.type === TokenType.LTE || t.type === TokenType.GTE || + t.type === TokenType.EQ || t.type === TokenType.NEQ) { + const op = advance().value; + const right = parseAddition(); + return { type: 'binary', op, left, right }; + } + return left; + } finally { exit(); } + } + + function parseAddition(): ASTNode { + enter(); + try { + let left = parseMultiplication(); + while (peek().type === TokenType.PLUS || peek().type === TokenType.MINUS) { + const op = advance().value; + const right = parseMultiplication(); + left = { type: 'binary', op, left, right }; + } + return left; + } finally { exit(); } + } + + function parseMultiplication(): ASTNode { + enter(); + try { + let left = parsePipe(); + while (peek().type === TokenType.STAR || peek().type === TokenType.SLASH) { + const op = advance().value; + const right = parsePipe(); + left = { type: 'binary', op, left, right }; + } + return left; + } finally { exit(); } + } + + function parsePipe(): ASTNode { + enter(); + try { + let left = parsePrimary(); + while (peek().type === TokenType.PIPE) { + advance(); + const filterName = expect(TokenType.IDENTIFIER).value; + let arg: ASTNode | undefined; + if (peek().type === TokenType.COLON) { + advance(); + arg = parsePrimary(); + } + left = { type: 'pipe', value: left, filter: filterName, arg }; + } + return left; + } finally { exit(); } + } + + function parsePrimary(): ASTNode { + enter(); + try { + const t = peek(); + + // Number + if (t.type === TokenType.NUMBER) { + advance(); + return { type: 'number', value: Number.parseFloat(t.value) }; + } + + // String + if (t.type === TokenType.STRING) { + advance(); + return { type: 'string', value: t.value }; + } + + // Boolean + if (t.type === TokenType.BOOLEAN) { + advance(); + return { type: 'boolean', value: t.value.toLowerCase() === 'true' }; + } + + // Array literal [a, b, c] + if (t.type === TokenType.LBRACKET) { + advance(); + const elements: ASTNode[] = []; + if (peek().type !== TokenType.RBRACKET) { + elements.push(parseExpr()); + while (peek().type === TokenType.COMMA) { + advance(); + elements.push(parseExpr()); + } + } + expect(TokenType.RBRACKET); + return { type: 'array', elements }; + } + + // Grouped expression + if (t.type === TokenType.LPAREN) { + advance(); + const inner = parseExpr(); + expect(TokenType.RPAREN); + return inner; + } + + // Identifier, field access, or function call + if (t.type === TokenType.IDENTIFIER) { + return parseCallOrField(); + } + + throw new ParseError(`Unexpected token '${t.value}'`, t.pos); + } finally { exit(); } + } + + function parseCallOrField(): ASTNode { + const name = advance().value; + + // Function call: name(args) + if (peek().type === TokenType.LPAREN) { + advance(); + const args: ASTNode[] = []; + if (peek().type !== TokenType.RPAREN) { + args.push(parseExpr()); + while (peek().type === TokenType.COMMA) { + advance(); + args.push(parseExpr()); + } + } + expect(TokenType.RPAREN); + let result: ASTNode = { type: 'call', name, args }; + // Allow chaining: func().field + while (peek().type === TokenType.DOT) { + advance(); + const field = expect(TokenType.IDENTIFIER).value; + result = { type: 'field_access', object: result, field }; + } + return result; + } + + // Field access: a.b.c + let node: ASTNode = { type: 'identifier', name }; + while (peek().type === TokenType.DOT) { + advance(); + const field = expect(TokenType.IDENTIFIER).value; + node = { type: 'field_access', object: node, field }; + } + return node; + } + + const ast = parseExpr(); + if (peek().type !== TokenType.EOF) { + throw new ParseError(`Unexpected trailing token '${peek().value}'`, peek().pos); + } + return ast; +} diff --git a/packages/ai-coach-analytics/src/dsl/safe-regex.ts b/packages/ai-coach-analytics/src/dsl/safe-regex.ts new file mode 100644 index 0000000..18e7d99 --- /dev/null +++ b/packages/ai-coach-analytics/src/dsl/safe-regex.ts @@ -0,0 +1,333 @@ +/*--------------------------------------------------------------------------------------------- + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the MIT License. See LICENSE in the project root for license information. + *--------------------------------------------------------------------------------------------*/ + +import { warnCore } from '../log'; + +/** Max pattern length. Anything longer is almost certainly a mistake. */ +const MAX_PATTERN_LEN = 1000; + +/** Max input length for `.test()`. Long enough for any realistic message body. */ +export const MAX_TEST_INPUT_LEN = 100_000; + +/** LRU-ish cache of compiled regexes (FIFO eviction). */ +const CACHE_MAX = 256; +const regexCache = new Map(); + +/** Patterns already seen and rejected, to avoid re-warning on every row. */ +const rejectedPatterns = new Set(); + +export function compileSafe(pattern: string, flags = ''): RegExp | null { + const key = `${pattern}::${flags}`; + const cached = regexCache.get(key); + if (cached) return cached; + if (rejectedPatterns.has(key)) return null; + + if (typeof pattern !== 'string' || pattern.length > MAX_PATTERN_LEN) { + rejectedPatterns.add(key); + warnCore('SafeRegex', `Pattern rejected (length > ${MAX_PATTERN_LEN}): ${pattern.slice(0, 40)}...`); + return null; + } + + if (!isLikelySafe(pattern)) { + rejectedPatterns.add(key); + warnCore('SafeRegex', `Pattern rejected (potential catastrophic backtracking): ${pattern}`); + return null; + } + + let re: RegExp; + try { + re = new RegExp(pattern, flags); + } catch (err) { + rejectedPatterns.add(key); + warnCore('SafeRegex', `Invalid regex '${pattern}' flags='${flags}': ${(err as Error).message}`); + return null; + } + + if (regexCache.size >= CACHE_MAX) { + // Evict oldest entry (insertion order). + const firstKey = regexCache.keys().next().value; + if (firstKey !== undefined) regexCache.delete(firstKey); + } + regexCache.set(key, re); + return re; +} + +export function testSafe(re: RegExp | null, input: string): boolean { + if (!re) return false; + const s = input.length > MAX_TEST_INPUT_LEN ? input.slice(0, MAX_TEST_INPUT_LEN) : input; + try { + // Reset lastIndex in case the pattern has the `g` flag. + re.lastIndex = 0; + return re.test(s); + } catch { + return false; + } +} + +/** Clear the compiled-regex cache (for tests / diagnostics). */ +export function clearRegexCache(): void { + regexCache.clear(); + rejectedPatterns.clear(); +} + +interface RegexGroupState { + start: number; + hasInnerUnbounded: boolean; +} + +function skipEscapedChar(pattern: string, index: number): number { + return pattern[index] === '\\' ? index + 1 : index; +} + +function skipCharacterClass(pattern: string, index: number): number { + let nextIndex = index + 1; + while (nextIndex < pattern.length && pattern[nextIndex] !== ']') { + if (pattern[nextIndex] === '\\') nextIndex++; + nextIndex++; + } + return nextIndex; +} + +function hasUnboundedQuantifierAt(pattern: string, index: number): boolean { + const char = pattern[index]; + return char === '+' || char === '*' || (char === '{' && /^\{\d+,\d*\}/.test(pattern.slice(index))); +} + +function closeGroup( + pattern: string, + index: number, + groupStack: RegexGroupState[], + starHeight: number, + maxStarHeight: number, +): { safe: boolean; starHeight: number; maxStarHeight: number } { + const group = groupStack.pop(); + if (!hasUnboundedQuantifierAt(pattern, index + 1)) { + return { safe: true, starHeight, maxStarHeight }; + } + + const nextStarHeight = starHeight + 1; + const nextMaxStarHeight = Math.max(maxStarHeight, nextStarHeight); + if (group?.hasInnerUnbounded) return { safe: false, starHeight: nextStarHeight, maxStarHeight: nextMaxStarHeight }; + + const body = pattern.slice(group ? group.start + 1 : 0, index); + if (hasOverlappingAlternation(body)) { + return { safe: false, starHeight: nextStarHeight, maxStarHeight: nextMaxStarHeight }; + } + + // A body that can match the empty string under an unbounded quantifier + // (e.g. `(a?b?)*`) is ambiguous and backtracks super-linearly even though + // no inner quantifier is itself unbounded. + if (isNullableBody(body)) { + return { safe: false, starHeight: nextStarHeight, maxStarHeight: nextMaxStarHeight }; + } + + return { safe: true, starHeight: nextStarHeight, maxStarHeight: nextMaxStarHeight }; +} + +function markInnerUnbounded(groupStack: RegexGroupState[]): void { + const top = groupStack[groupStack.length - 1]; + if (top) top.hasInnerUnbounded = true; +} + +export function isLikelySafe(pattern: string): boolean { + let starHeight = 0; + let maxStarHeight = 0; + const groupStack: RegexGroupState[] = []; + + for (let i = 0; i < pattern.length; i++) { + const ch = pattern[i]; + + if (ch === '\\') { + i = skipEscapedChar(pattern, i); + continue; + } + if (ch === '[') { + i = skipCharacterClass(pattern, i); + continue; + } + if (ch === '(') { + groupStack.push({ start: i, hasInnerUnbounded: false }); + continue; + } + if (ch === ')') { + const result = closeGroup(pattern, i, groupStack, starHeight, maxStarHeight); + if (!result.safe) return false; + starHeight = result.starHeight; + maxStarHeight = result.maxStarHeight; + continue; + } + if (hasUnboundedQuantifierAt(pattern, i)) { + markInnerUnbounded(groupStack); + } + } + + return maxStarHeight <= 2; +} + +const CLASS_ESCAPES = new Set(['w', 'W', 'd', 'D', 's', 'S']); + +function splitTopLevelBranches(body: string): string[] { + const branches: string[] = []; + let depth = 0; + let start = 0; + for (let i = 0; i < body.length; i++) { + const ch = body[i]; + if (ch === '\\') { i++; continue; } + if (ch === '[') { i = skipCharacterClass(body, i); continue; } + if (ch === '(') depth++; + else if (ch === ')') depth--; + else if (ch === '|' && depth === 0) { + branches.push(body.slice(start, i)); + start = i + 1; + } + } + branches.push(body.slice(start)); + return branches; +} + +// Alternation branches that can match the same first character cause +// catastrophic backtracking under an unbounded quantifier (e.g. `(.|a)+`). +function hasOverlappingAlternation(body: string): boolean { + const tokens = splitTopLevelBranches(body).map(firstToken); + if (tokens.length < 2) return false; + + for (let i = 0; i < tokens.length; i++) { + for (let j = i + 1; j < tokens.length; j++) { + if (tokensOverlap(tokens[i], tokens[j])) return true; + } + } + return false; +} + +// A branch's leading token: either a literal run or a single-char matcher +// (`.`, a class escape, or a `[...]` class). `null` means indeterminate. +type FirstToken = + | { set: false; literal: string } + | { set: true; source: string } + | null; + +function firstToken(branch: string): FirstToken { + const body = branch.startsWith('^') ? branch.slice(1) : branch; + if (body.length === 0) return null; + + if (body[0] === '.') return { set: true, source: '.' }; + if (body[0] === '[') return { set: true, source: body.slice(0, skipCharacterClass(body, 0) + 1) }; + if (body[0] === '\\' && body.length > 1 && CLASS_ESCAPES.has(body[1])) { + return { set: true, source: body.slice(0, 2) }; + } + + const prefix = literalPrefix(body); + return prefix ? { set: false, literal: prefix } : null; +} + +function tokensOverlap(a: FirstToken, b: FirstToken): boolean { + if (!a || !b) return false; + if (!a.set && !b.set) return a.literal.startsWith(b.literal) || b.literal.startsWith(a.literal); + if (a.set && b.set) return true; + const set = a.set ? a : (b as Extract); + const literal = a.set ? (b as Extract) : a; + return setMatchesChar(set.source, literal.literal[0]); +} + +// Whether a single-char matcher source (e.g. `[a-z]`, `\w`, `.`) accepts `ch`. +// An unparseable source is treated as a match (fail closed toward rejection). +function setMatchesChar(source: string, ch: string): boolean { + try { + return new RegExp(`^(?:${source})$`).test(ch); + } catch { + return true; + } +} + +function literalPrefix(branch: string): string { + let out = ''; + for (let i = 0; i < branch.length; i++) { + const ch = branch[i]; + if (ch === '\\' && i + 1 < branch.length) { + out += branch[i + 1]; + i++; + continue; + } + if ('()[]{}|+*?.^$'.includes(ch)) break; + out += ch; + } + return out; +} + +/* ---- Nullable-body detection ---- */ + +/** + * Whether a group body can match the empty string. Used by `closeGroup`: + * an unbounded quantifier over a nullable body (e.g. `(a?b?)*`) creates + * ambiguity that backtracks super-linearly. Conservative: returns true when + * ANY top-level branch is nullable, and treats lookaround groups (which are + * zero-width, so quantifying them is always degenerate) as nullable. + */ +function isNullableBody(body: string): boolean { + let b = body; + if (b.startsWith('?:')) { + b = b.slice(2); + } else if (b.startsWith('?<') && !b.startsWith('?<=') && !b.startsWith('?'); + if (close === -1) return true; + b = b.slice(close + 1); + } else if (b.startsWith('?')) { + return true; + } + return splitTopLevelBranches(b).some(isNullableBranch); +} + +/** Whether every atom in the branch is optional (or the branch is empty). */ +function isNullableBranch(branch: string): boolean { + let i = 0; + while (i < branch.length) { + const ch = branch[i]; + if (ch === '^' || ch === '$') { i++; continue; } // zero-width anchors + let atomEnd = i; + let atomNullable = false; + if (ch === '\\') { + atomEnd = i + 1; + } else if (ch === '[') { + atomEnd = skipCharacterClass(branch, i); + } else if (ch === '(') { + atomEnd = findGroupEnd(branch, i); + atomNullable = isNullableBody(branch.slice(i + 1, atomEnd)); + } + let next = atomEnd + 1; + const q = branch[next]; + if (q === '?' || q === '*') { + atomNullable = true; + next++; + } else if (q === '+') { + next++; + } else if (q === '{') { + const m = /^\{(\d+),?\d*\}/.exec(branch.slice(next)); + if (m) { + if (m[1] === '0') atomNullable = true; + next += m[0].length; + } + } + if (branch[next] === '?') next++; // lazy-quantifier modifier + if (!atomNullable) return false; + i = next; + } + return true; +} + +/** Index of the `)` closing the group that opens at `start`. */ +function findGroupEnd(s: string, start: number): number { + let depth = 0; + for (let i = start; i < s.length; i++) { + const ch = s[i]; + if (ch === '\\') { i++; continue; } + if (ch === '[') { i = skipCharacterClass(s, i); continue; } + if (ch === '(') depth++; + else if (ch === ')') { + depth--; + if (depth === 0) return i; + } + } + return s.length; +} diff --git a/packages/ai-coach-analytics/src/dsl/schema.ts b/packages/ai-coach-analytics/src/dsl/schema.ts new file mode 100644 index 0000000..9911d5b --- /dev/null +++ b/packages/ai-coach-analytics/src/dsl/schema.ts @@ -0,0 +1,209 @@ +/*--------------------------------------------------------------------------------------------- + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the MIT License. See LICENSE in the project root for license information. + *--------------------------------------------------------------------------------------------*/ + +/** + * Field schema for SessionRequest and Session types. + * Used by the Data Explorer and Rule Playground to show available fields. + * Also used by the NL compiler to generate valid DSL expressions. + */ + +export interface FieldInfo { + name: string; + type: 'number' | 'string' | 'boolean' | 'string[]' | 'object' | 'object[]' | 'number | null'; + description: string; + scope: 'request' | 'session' | 'both'; + example?: string; +} + +export const FIELD_SCHEMA: FieldInfo[] = [ + // SessionRequest fields + { name: 'requestId', type: 'string', description: 'Unique request identifier', scope: 'request' }, + { name: 'timestamp', type: 'number | null', description: 'Request timestamp (epoch ms)', scope: 'request', example: '1713216000000' }, + { name: 'messageText', type: 'string', description: 'User message text', scope: 'request' }, + { name: 'responseText', type: 'string', description: 'AI response text', scope: 'request' }, + { name: 'isCanceled', type: 'boolean', description: 'Whether the request was canceled', scope: 'request' }, + { name: 'agentName', type: 'string', description: 'Agent name (copilot, custom agent, etc.)', scope: 'request' }, + { name: 'agentMode', type: 'string', description: 'Agent mode: agent, ask, edit, plan, or custom agent name', scope: 'request', example: '"agent"' }, + { name: 'modelId', type: 'string', description: 'Model identifier (gpt-4.1, claude-sonnet, etc.)', scope: 'request', example: '"gpt-4.1"' }, + { name: 'toolsUsed', type: 'string[]', description: 'List of tools used in this request', scope: 'request' }, + { name: 'editedFiles', type: 'string[]', description: 'Files edited during this request', scope: 'request' }, + { name: 'referencedFiles', type: 'string[]', description: 'Files referenced via #file or @workspace', scope: 'request' }, + { name: 'slashCommand', type: 'string', description: 'Slash command used (empty if none)', scope: 'request' }, + { name: 'variableKinds', type: 'object', description: 'Record of variable kinds used', scope: 'request', example: '{ "file": 2 }' }, + { name: 'customInstructions', type: 'string[]', description: 'Custom instruction files active', scope: 'request' }, + { name: 'skillsUsed', type: 'string[]', description: 'Skills invoked during this request', scope: 'request' }, + { name: 'firstProgress', type: 'number | null', description: 'Time to first progress event (ms)', scope: 'request' }, + { name: 'totalElapsed', type: 'number | null', description: 'Total response time (ms)', scope: 'request' }, + { name: 'messageLength', type: 'number', description: 'User message length in characters', scope: 'request', example: '45' }, + { name: 'responseLength', type: 'number', description: 'AI response length in characters', scope: 'request' }, + { name: 'userCode', type: 'object[]', description: 'Code blocks in user message [{language, loc}]', scope: 'request' }, + { name: 'aiCode', type: 'object[]', description: 'Code blocks in AI response [{language, loc}]', scope: 'request' }, + { name: 'toolConfirmations', type: 'object[]', description: 'Tool confirmation events', scope: 'request' }, + { name: 'promptTokens', type: 'number | null', description: 'Prompt token count (total input context size, includes cached portion)', scope: 'request' }, + { name: 'completionTokens', type: 'number | null', description: 'Completion token count', scope: 'request' }, + { name: 'cacheReadTokens', type: 'number | null', description: 'Cached tokens read (subset of promptTokens)', scope: 'request' }, + { name: 'cacheWriteTokens', type: 'number | null', description: 'Cached tokens written / cache creation', scope: 'request' }, + { name: 'reasoningEffort', type: 'string', description: 'Reasoning effort level (max | high | medium | low) when known', scope: 'request', example: '"high"' }, + { name: 'workType', type: 'string', description: 'Work type classification: feature, bugfix, refactor, etc.', scope: 'request' }, + + // Session fields + { name: 'sessionId', type: 'string', description: 'Unique session identifier', scope: 'session' }, + { name: 'workspaceId', type: 'string', description: 'Workspace identifier', scope: 'session' }, + { name: 'workspaceName', type: 'string', description: 'Workspace display name', scope: 'session' }, + { name: 'location', type: 'string', description: 'Workspace file path', scope: 'session' }, + { name: 'harness', type: 'string', description: 'IDE harness: VS Code, Local Agent, Xcode, etc.', scope: 'session', example: '"VS Code"' }, + { name: 'creationDate', type: 'number | null', description: 'Session creation timestamp (epoch ms)', scope: 'session' }, + { name: 'lastMessageDate', type: 'number | null', description: 'Timestamp of last message (epoch ms)', scope: 'session' }, + { name: 'requestCount', type: 'number', description: 'Number of requests in the session', scope: 'session', example: '12' }, + { name: 'requests', type: 'object[]', description: 'Array of SessionRequest objects', scope: 'session' }, + { name: 'hasDevcontainer', type: 'boolean', description: 'Session ran in a devcontainer (detected from runtime /workspaces/ paths)', scope: 'session' }, + { name: 'customInstructionsBytes', type: 'number', description: 'Bytes in .github/copilot-instructions.md (0 if absent)', scope: 'session', example: '4200' }, +]; + +/** Built-in named metric primitives that can be referenced in .metric.md or .rule.md files. */ +export interface MetricPrimitive { + id: string; + name: string; + description: string; + scope: 'requests' | 'sessions'; + /** DSL filter expression */ + filter: string; + /** How to aggregate: ratio, count, sum(field), avg(field), etc. */ + aggregation: string; + /** Example template for matching items */ + exampleTemplate?: string; +} + +export const METRIC_PRIMITIVES: MetricPrimitive[] = [ + { + id: 'cancel_rate', + name: 'Cancellation Rate', + description: 'Ratio of canceled requests to total requests', + scope: 'requests', + filter: 'isCanceled == true', + aggregation: 'ratio', + }, + { + id: 'avg_message_length', + name: 'Average Message Length', + description: 'Average character length of user messages', + scope: 'requests', + filter: 'messageLength > 0', + aggregation: 'avg(messageLength)', + }, + { + id: 'avg_response_time', + name: 'Average Response Time', + description: 'Average total elapsed time in milliseconds', + scope: 'requests', + filter: 'totalElapsed > 0', + aggregation: 'avg(totalElapsed)', + }, + { + id: 'unique_models', + name: 'Unique Models Used', + description: 'Count of distinct model IDs', + scope: 'requests', + filter: 'modelId != ""', + aggregation: 'unique(modelId)', + }, + { + id: 'files_per_request', + name: 'File References per Request', + description: 'Average number of referenced files per request', + scope: 'requests', + filter: 'messageLength > 0', + aggregation: 'avg(referencedFiles.length)', + }, + { + id: 'session_length', + name: 'Session Length', + description: 'Average number of requests per session', + scope: 'sessions', + filter: 'requestCount > 0', + aggregation: 'avg(requestCount)', + }, + { + id: 'tool_usage_rate', + name: 'Tool Usage Rate', + description: 'Ratio of requests that used at least one tool', + scope: 'requests', + filter: 'toolsUsed.length > 0', + aggregation: 'ratio', + }, + { + id: 'agent_mode_rate', + name: 'Agent Mode Usage', + description: 'Ratio of requests using agentic modes (agent, plan, edit, custom)', + scope: 'requests', + filter: 'agentMode != "" && agentMode != "chat" && agentMode != "ask"', + aggregation: 'ratio', + }, + { + id: 'code_output_rate', + name: 'Code Output Rate', + description: 'Ratio of requests that produced code blocks', + scope: 'requests', + filter: 'aiCode.length > 0', + aggregation: 'ratio', + }, + { + id: 'custom_instructions_rate', + name: 'Custom Instructions Rate', + description: 'Ratio of requests with custom instructions active', + scope: 'requests', + filter: 'customInstructions.length > 0', + aggregation: 'ratio', + }, +]; + +/** Available DSL functions for documentation and autocomplete. */ +export interface FunctionInfo { + name: string; + signature: string; + description: string; + category: 'string' | 'math' | 'date' | 'array' | 'object' | 'utility'; +} + +export const FUNCTION_CATALOG: FunctionInfo[] = [ + { name: 'length', signature: 'length(x)', description: 'Array or string length', category: 'string' }, + { name: 'contains', signature: 'contains(s, sub)', description: 'String contains substring', category: 'string' }, + { name: 'startsWith', signature: 'startsWith(s, sub)', description: 'String starts with prefix', category: 'string' }, + { name: 'endsWith', signature: 'endsWith(s, sub)', description: 'String ends with suffix', category: 'string' }, + { name: 'matches', signature: 'matches(s, /re/)', description: 'Regex match', category: 'string' }, + { name: 'lower', signature: 'lower(s)', description: 'To lower case', category: 'string' }, + { name: 'upper', signature: 'upper(s)', description: 'To upper case', category: 'string' }, + { name: 'trim', signature: 'trim(s)', description: 'Strip whitespace', category: 'string' }, + { name: 'truncate', signature: 'truncate(s, n)', description: 'Truncate to n chars', category: 'string' }, + { name: 'split', signature: 'split(s, sep)', description: 'Split into array', category: 'string' }, + { name: 'join', signature: 'join(arr, sep)', description: 'Join array to string', category: 'string' }, + { name: 'abs', signature: 'abs(n)', description: 'Absolute value', category: 'math' }, + { name: 'floor', signature: 'floor(n)', description: 'Floor', category: 'math' }, + { name: 'ceil', signature: 'ceil(n)', description: 'Ceiling', category: 'math' }, + { name: 'round', signature: 'round(n)', description: 'Round', category: 'math' }, + { name: 'min', signature: 'min(a, b)', description: 'Minimum of two values', category: 'math' }, + { name: 'max', signature: 'max(a, b)', description: 'Maximum of two values', category: 'math' }, + { name: 'hour', signature: 'hour(ts)', description: 'Extract hour (0-23) from ms', category: 'date' }, + { name: 'dayOfWeek', signature: 'dayOfWeek(ts)', description: 'Day of week (0=Sun)', category: 'date' }, + { name: 'month', signature: 'month(ts)', description: 'Month (0=Jan)', category: 'date' }, + { name: 'year', signature: 'year(ts)', description: 'Full year', category: 'date' }, + { name: 'includes', signature: 'includes(arr, val)', description: 'Array includes value', category: 'array' }, + { name: 'some', signature: 'some(arr, field)', description: 'Any element has truthy field', category: 'array' }, + { name: 'count', signature: 'count(arr)', description: 'Array length', category: 'array' }, + { name: 'sum', signature: 'sum(arr, field)', description: 'Sum numeric field across array',category: 'array' }, + { name: 'avg', signature: 'avg(arr, field)', description: 'Average numeric field', category: 'array' }, + { name: 'unique', signature: 'unique(arr, field)', description: 'Count distinct values', category: 'array' }, + { name: 'flatUnique', signature: 'flatUnique(arr, field)', description: 'Count distinct values across sub-arrays', category: 'array' }, + { name: 'reasoningEffortStats', signature: 'reasoningEffortStats(reqs[, level])', description: 'Premium reasoning-effort usage ratio (level: "high"|"max")', category: 'array' }, + { name: 'instructionBloatStats', signature: 'instructionBloatStats(sessions[, maxBytes])', description: 'Custom-instructions size analysis per workspace', category: 'array' }, + + { name: 'excessFileContextStats', signature: 'excessFileContextStats(reqs[, minFiles])', description: 'Outlier requests with huge referencedFiles arrays', category: 'array' }, + { name: 'hasSkillByPattern', signature: 'hasSkillByPattern(reqs, /pat/)', description: '1 if any skillsUsed entry matches the regex', category: 'array' }, + { name: 'keys', signature: 'keys(obj)', description: 'Object keys', category: 'object' }, + { name: 'values', signature: 'values(obj)', description: 'Object values', category: 'object' }, + { name: 'has', signature: 'has(obj, key)', description: 'Object has key', category: 'object' }, + { name: 'coalesce', signature: 'coalesce(a, b, ...)',description: 'First non-null value', category: 'utility' }, + { name: 'iif', signature: 'iif(cond, t, f)', description: 'Conditional expression', category: 'utility' }, +]; diff --git a/packages/ai-coach-analytics/src/dsl/types.ts b/packages/ai-coach-analytics/src/dsl/types.ts new file mode 100644 index 0000000..07a4f5a --- /dev/null +++ b/packages/ai-coach-analytics/src/dsl/types.ts @@ -0,0 +1,108 @@ +/*--------------------------------------------------------------------------------------------- + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the MIT License. See LICENSE in the project root for license information. + *--------------------------------------------------------------------------------------------*/ + +/** + * AST and token types for the metric expression DSL. + * + * The DSL supports: + * - Field access: messageLength, referencedFiles.length, variableKinds.file + * - Comparisons: <, >, <=, >=, ==, != + * - Logical ops: AND, OR, NOT + * - Functions: length(), contains(), matches(), hour(), dayOfWeek(), ... + * - Literals: 42, 3.14, "hello", true, false + * - Pipe filters: messageText | truncate:80 + */ + +/* ------------------------------------------------------------------ */ +/* Tokens */ +/* ------------------------------------------------------------------ */ + +export enum TokenType { + NUMBER, + STRING, + BOOLEAN, + IDENTIFIER, + // Comparison operators + LT, + GT, + LTE, + GTE, + EQ, + NEQ, + // Logical operators + AND, + OR, + NOT, + // Arithmetic operators + PLUS, + MINUS, + STAR, + SLASH, + // Punctuation + LPAREN, + RPAREN, + LBRACKET, + RBRACKET, + DOT, + COMMA, + PIPE, + COLON, + // End + EOF, +} + +export interface Token { + type: TokenType; + value: string; + pos: number; +} + +/* ------------------------------------------------------------------ */ +/* AST Nodes */ +/* ------------------------------------------------------------------ */ + +export interface NumberNode { type: 'number'; value: number } +export interface StringNode { type: 'string'; value: string } +export interface BooleanNode { type: 'boolean'; value: boolean } +export interface IdentifierNode { type: 'identifier'; name: string } +export interface FieldAccessNode { type: 'field_access'; object: ASTNode; field: string } +export interface BinaryNode { type: 'binary'; op: string; left: ASTNode; right: ASTNode } +export interface UnaryNode { type: 'unary'; op: string; operand: ASTNode } +export interface CallNode { type: 'call'; name: string; args: ASTNode[] } +export interface PipeNode { type: 'pipe'; value: ASTNode; filter: string; arg?: ASTNode } +export interface ArrayNode { type: 'array'; elements: ASTNode[] } + +export type ASTNode = + | NumberNode + | StringNode + | BooleanNode + | IdentifierNode + | FieldAccessNode + | BinaryNode + | UnaryNode + | CallNode + | PipeNode + | ArrayNode; + +/* ------------------------------------------------------------------ */ +/* Metric definition (parsed from .metric.md or inline # Filter) */ +/* ------------------------------------------------------------------ */ + +export type AggregationType = 'count' | 'ratio' | 'sum' | 'avg' | 'min' | 'max' | 'unique' | 'percentile'; + +export interface MetricAggregation { + type: AggregationType; + field?: string; // for sum/avg/min/max/unique + percentile?: number; // for percentile(field, N) +} + +/* ------------------------------------------------------------------ */ +/* Compiled metric / filter / trigger */ +/* ------------------------------------------------------------------ */ + +export type FilterFn = (row: Record) => boolean; +export type MetricFn = (rows: Record[]) => number; +export type TriggerFn = (emission: { count: number; total: number; ratio: number; extra: Record }) => boolean; +export type FormatFn = (row: Record) => string; diff --git a/packages/ai-coach-analytics/src/helpers.ts b/packages/ai-coach-analytics/src/helpers.ts new file mode 100644 index 0000000..62918dd --- /dev/null +++ b/packages/ai-coach-analytics/src/helpers.ts @@ -0,0 +1,296 @@ +/*--------------------------------------------------------------------------------------------- + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the MIT License. See LICENSE in the project root for license information. + *--------------------------------------------------------------------------------------------*/ + +/* Shared date and model helpers used across analyzer modules */ + +import { WorkType } from './types'; +import { MODEL_MULTIPLIERS, MODEL_TOKEN_RATES } from './constants'; + +/* ---- File-URI helper ---- */ + +/** decodeURIComponent that returns the input unchanged on malformed sequences (e.g. a literal `%` in a path). */ +function safeDecode(s: string): string { + try { return decodeURIComponent(s); } catch { return s; } +} + +/** + * Convert a `file://` URI to a local filesystem path, handling Windows drive-letter + * URIs (e.g. `file:///C:/Users/...`) correctly. Falls back to decoding + * for non-URI strings so callers don't need to branch. + */ +export function fileUriToPath(raw: string): string { + if (!raw.startsWith('file://')) return safeDecode(raw); + // Strip scheme: file:///C:/foo → /C:/foo file:///home/u → /home/u + let p = safeDecode(raw.replace(/^file:\/\//, '')); + // On Windows the URI has an extra leading slash before the drive letter: /C:/... + if (/^\/[A-Za-z]:/.test(p)) p = p.slice(1); + return p; +} + +/* ---- Date helpers ---- */ +export function toDateStr(ts: number): string { + const d = new Date(ts); + return `${d.getFullYear()}-${String(d.getMonth() + 1).padStart(2, '0')}-${String(d.getDate()).padStart(2, '0')}`; +} + +export function startOfDay(ts: number): number { + const d = new Date(ts); + d.setHours(0, 0, 0, 0); + return d.getTime(); +} + +export function endOfDay(ts: number): number { + const d = new Date(ts); + d.setHours(23, 59, 59, 999); + return d.getTime(); +} + +export function isoWeek(d: Date): string { + const thu = new Date(d); + thu.setDate(d.getDate() - ((d.getDay() + 6) % 7) + 3); + const yr = thu.getFullYear(); + const wk = Math.ceil((((thu.getTime() - new Date(yr, 0, 4).getTime()) / 86400000) + new Date(yr, 0, 4).getDay() + 1) / 7); + return `${yr}-W${String(wk).padStart(2, '0')}`; +} + +/** Return every YYYY-MM-DD between the first and last entry in `days` (inclusive). */ +export function fillDayRange(days: string[]): string[] { + if (days.length <= 1) return days; + const sorted = [...days].sort(); + const start = new Date(sorted[0] + 'T00:00:00'); + const end = new Date(sorted[sorted.length - 1] + 'T00:00:00'); + const result: string[] = []; + for (const d = new Date(start); d <= end; d.setDate(d.getDate() + 1)) { + result.push(toDateStr(d.getTime())); + } + return result; +} + +/** Return every ISO week label between the first and last entry in `weeks` (inclusive). */ +export function fillWeekRange(weeks: string[]): string[] { + if (weeks.length <= 1) return weeks; + const sorted = [...weeks].sort(); + const toMonday = (label: string): Date => { + const [y, w] = label.split('-W').map(Number); + const jan4 = new Date(y, 0, 4); + const dow = (jan4.getDay() + 6) % 7; + const mon = new Date(jan4); + mon.setDate(jan4.getDate() - dow + (w - 1) * 7); + return mon; + }; + const start = toMonday(sorted[0]); + const end = toMonday(sorted[sorted.length - 1]); + const result: string[] = []; + for (const d = new Date(start); d <= end; d.setDate(d.getDate() + 7)) { + result.push(isoWeek(d)); + } + return result; +} + +/** Return every YYYY-MM between the first and last entry in `months` (inclusive). */ +export function fillMonthRange(months: string[]): string[] { + if (months.length <= 1) return months; + const sorted = [...months].sort(); + const [sy, sm] = sorted[0].split('-').map(Number); + const [ey, em] = sorted[sorted.length - 1].split('-').map(Number); + const result: string[] = []; + for (let y = sy, m = sm; y < ey || (y === ey && m <= em); m++) { + if (m > 12) { m = 1; y++; } + result.push(`${y}-${String(m).padStart(2, '0')}`); + } + return result; +} + +/* ---- Model helpers ---- */ + +/** Effort suffixes that get baked into model IDs by the GitHub model router + * (e.g. `claude-opus-4.7-high`, `claude-opus-4.7-xhigh`). We strip these + * from the normalized model name so cost/context analyses group by family, + * while the effort value is captured separately via extractReasoningEffortFromModelId. */ +const EFFORT_SUFFIX_RE = /-(xhigh|extra-high|max|high|medium|med|low|minimal)$/; + +export function normalizeModel(modelId: string): string { + let m = modelId.trim(); + for (const prefix of ['copilot/', 'github.copilot-chat/', 'github/']) { + if (m.startsWith(prefix)) { m = m.slice(prefix.length); break; } + } + m = m.replace(/-thought$/, '').replace(/-preview$/, '').replace(/-latest$/, ''); + // Claude's API returns hyphenated version numbers (claude-opus-4-6) where + // our rate tables use dots (claude-opus-4.6). Also strip -YYYYMMDD date + // suffixes that older API responses append (claude-haiku-4-5-20251001). + // Non-global replace: only the first digit-hyphen-digit is converted, + // preserving suffixes like -1m. + if (m.startsWith('claude-')) { + m = m.replace(/-\d{8}$/, ''); + m = m.replace(/(\d)-(\d)/, '$1.$2'); + } + // Strip effort suffixes ONLY for known reasoning-capable families (avoid + // false positives on models like gpt-5.4-mini or gemini-3-flash). + if (EFFORT_BEARING_RE.test(m) && !NON_EFFORT_SUFFIXES_RE.test(m)) { + m = m.replace(EFFORT_SUFFIX_RE, ''); + } + return m.trim(); +} + +export function modelMultiplier(modelId: string): number { + const key = normalizeModel(modelId); + if (MODEL_MULTIPLIERS[key] !== undefined) return MODEL_MULTIPLIERS[key]; + for (const [k, v] of Object.entries(MODEL_MULTIPLIERS)) { + if (key.startsWith(k)) return v; + } + return 1; +} + +/* ---- Reasoning effort inference ---- */ + +/** Regex matching reasoning-capable model families. Uses prefix patterns so + * future point-releases (e.g. `claude-opus-4.8`, `gpt-5.7`) are covered + * automatically. Allowlist-based to avoid false-positives on suffixes that + * don't denote effort (e.g. `-fast`, `-1m`, `-mini`). */ +const EFFORT_BEARING_RE = /^(?:claude-opus-4|claude-sonnet-4|gpt-5|o[1-9]|o\d{2,}|gemini-[2-9]|gemini-\d{2,})/; + +/** Suffixes that look like effort markers but are NOT reasoning-effort levels. + * These are excluded before matching effort suffixes to prevent false positives. */ +const NON_EFFORT_SUFFIXES_RE = /-(fast|1m|mini|nano|micro|preview|latest|thought|turbo|flash|pro|ultra|internal)$/; + +/** Map a `-suffix` on a known reasoning-capable model id to a canonical + * effort level. Returns `null` for unknown families or unknown suffixes + * (do NOT guess — `null` propagates to "unknown" in the data). */ +export function extractReasoningEffortFromModelId( + modelId: string, +): 'max' | 'high' | 'medium' | 'low' | null { + if (!modelId) return null; + // Strip vendor prefix and legacy suffixes (-thought/-preview/-latest) but + // NOT effort suffixes — those are what we want to detect here. + let key = modelId.trim(); + for (const prefix of ['copilot/', 'github.copilot-chat/', 'github/']) { + if (key.startsWith(prefix)) { key = key.slice(prefix.length); break; } + } + key = key.replace(/-thought$/, '').replace(/-preview$/, '').replace(/-latest$/, '').toLowerCase(); + // Same Claude API normalization as normalizeModel (date suffix + version dots). + if (key.startsWith('claude-')) { + key = key.replace(/-\d{8}$/, ''); + key = key.replace(/(\d)-(\d)/, '$1.$2'); + } + if (!EFFORT_BEARING_RE.test(key)) return null; + if (NON_EFFORT_SUFFIXES_RE.test(key)) return null; + if (key.endsWith('-xhigh') || key.endsWith('-extra-high') || key.endsWith('-max')) return 'max'; + if (key.endsWith('-high')) return 'high'; + if (key.endsWith('-medium') || key.endsWith('-med')) return 'medium'; + if (key.endsWith('-low') || key.endsWith('-minimal')) return 'low'; + return null; +} + +/** Normalize free-form effort strings reported by harnesses into the canonical set. */ +export function canonicalizeReasoningEffort( + raw: string | null | undefined, +): 'max' | 'high' | 'medium' | 'low' | null { + if (!raw) return null; + const s = String(raw).toLowerCase().trim(); + if (s === 'max' || s === 'maximum' || s === 'xhigh' || s === 'extra-high' || s === 'extreme') return 'max'; + if (s === 'high') return 'high'; + if (s === 'medium' || s === 'med' || s === 'default') return 'medium'; + if (s === 'low' || s === 'minimal') return 'low'; + return null; +} + +/* ---- Native token resolution ---- */ + +/** + * Resolve token counts for a request. + * + * Returns *only* native counts reported by the harness. No char-based + * approximation is performed: when a side is missing the corresponding + * value is `0` and the side-specific flag (`hasInput` / `hasOutput`) is + * false. `missing` is `true` whenever **either** input OR output is absent + * — credit math requires both sides to be considered complete. + * + * `promptTokens` is the *total* input context size (includes the cached + * portion when known). `cacheReadTokens` and `cacheWriteTokens` are + * subsets used only for billing math; they default to 0 (treated as + * fully uncached) when not provided. + */ +export function resolveTokens( + promptTokens: number | null, + completionTokens: number | null, + cacheReadTokens: number | null = null, + cacheWriteTokens: number | null = null, +): { + input: number; + output: number; + cacheRead: number; + cacheWrite: number; + uncachedInput: number; + hasInput: boolean; + hasOutput: boolean; + missing: boolean; +} { + const hasInput = promptTokens != null; + const hasOutput = completionTokens != null; + const input = hasInput ? promptTokens : 0; + const output = hasOutput ? completionTokens : 0; + const cacheRead = cacheReadTokens ?? 0; + const cacheWrite = cacheWriteTokens ?? 0; + // Uncached input cannot be negative even if cache totals exceed prompt + // (which can happen with rounding from session-level totals). + const uncachedInput = Math.max(0, input - cacheRead - cacheWrite); + return { + input, + output, + cacheRead, + cacheWrite, + uncachedInput, + hasInput, + hasOutput, + missing: !hasInput || !hasOutput, + }; +} + +/** + * Calculate AI Credit cost for a single request or aggregated bucket. + * 1 AI Credit = $0.01 USD. Rates are in $/1M tokens. + * + * `inputTokens` is the *uncached* input portion (billed at the input rate). + * `cacheReadTokens` and `cacheWriteTokens` (defaulting to 0) are billed at + * their lower respective rates. Returns credits, not dollars. + */ +export function tokenCostInCredits( + model: string, + inputTokens: number, + outputTokens: number, + cacheReadTokens: number = 0, + cacheWriteTokens: number = 0, +): number { + const rates = MODEL_TOKEN_RATES[normalizeModel(model)]; + if (!rates) { + // Fallback: use model multiplier as rough proxy (1 PRU ≈ 1 credit) + return modelMultiplier(model); + } + const inputCost = (inputTokens / 1_000_000) * rates.input; + const outputCost = (outputTokens / 1_000_000) * rates.output; + const cacheReadCost = (cacheReadTokens / 1_000_000) * rates.cached; + const cacheWriteCost = (cacheWriteTokens / 1_000_000) * (rates.cacheWrite ?? rates.input); + // Convert USD to credits (1 credit = $0.01) + return (inputCost + outputCost + cacheReadCost + cacheWriteCost) * 100; +} + +/* ---- Work-type classification ---- */ +const WORK_PATTERNS: [RegExp, WorkType][] = [ + [/\b(fix|bug|error|issue|crash|exception|debug|problem|broken|fail|wrong)\b/i, 'bug fix'], + [/\b(refactor|rename|extract|move|cleanup|simplify|restructure|reorganize)\b/i, 'refactor'], + [/\b(review|pr|pull request|code review|comment on|feedback|approve)\b/i, 'code review'], + [/\b(test|spec|expect|assert|mock|stub|coverage|vitest|jest|pytest|unittest)\b/i, 'test'], + [/\b(doc|readme|comment|explain|jsdoc|typedoc|docstring|swagger|openapi)\b/i, 'docs'], + [/\b(style|css|scss|sass|theme|layout|padding|margin|font|color|design|ui)\b/i, 'style'], + [/\b(config|setup|install|dependency|package|ci|cd|pipeline|deploy|docker|k8s|terraform|bicep|env|yaml|yml)\b/i, 'config'], + [/\b(add|create|implement|build|feature|new|scaffold|generate|develop)\b/i, 'feature'], +]; + +export function classifyWorkType(msg: string): WorkType { + for (const [re, wt] of WORK_PATTERNS) { + if (re.test(msg)) return wt; + } + return 'other'; +} diff --git a/packages/ai-coach-analytics/src/index.ts b/packages/ai-coach-analytics/src/index.ts new file mode 100644 index 0000000..fc2585d --- /dev/null +++ b/packages/ai-coach-analytics/src/index.ts @@ -0,0 +1,8 @@ +export * from "./types"; + +export { findClaudeDirs, parseClaudeSessions, parseClaudeSessionsAsync } from "./parser-claude"; +export { findCodexDirs, parseCodexSessions } from "./parser-codex"; + +export { registerAllBuiltinRules, registerAllBuiltinMetrics } from "./rule-loader"; +export { getAllRules, getRulesGrouped } from "./rule-engine"; +export { runDetectors, runEmitters, getDetectorGroupCounts } from "./detector-registry"; diff --git a/packages/ai-coach-analytics/src/log.ts b/packages/ai-coach-analytics/src/log.ts new file mode 100644 index 0000000..c315803 --- /dev/null +++ b/packages/ai-coach-analytics/src/log.ts @@ -0,0 +1,68 @@ +/*--------------------------------------------------------------------------------------------- + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the MIT License. See LICENSE in the project root for license information. + *--------------------------------------------------------------------------------------------*/ + +/* Core logging helpers -- keep parser/analyzer diagnostics consistent. */ + +export type CoreLogLevel = 'debug' | 'info' | 'warn' | 'error'; + +export interface CoreLogEntry { + timestamp: number; + level: CoreLogLevel; + scope: string; + message: string; + detail?: string; +} + +const recentEntries: CoreLogEntry[] = []; +const MAX_RECENT_ENTRIES = 200; + +function detailToString(detail: unknown): string | undefined { + if (detail === undefined || detail === null) return undefined; + if (detail instanceof Error) return detail.message; + if (typeof detail === 'string') return detail; + if (typeof detail === 'number' || typeof detail === 'boolean' || typeof detail === 'bigint') { + return `${detail}`; + } + try { + return JSON.stringify(detail); + } catch { + return Object.prototype.toString.call(detail); + } +} + +function emit(level: CoreLogLevel, scope: string, message: string, detail?: unknown): void { + const entry: CoreLogEntry = { + timestamp: Date.now(), + level, + scope, + message, + detail: detailToString(detail), + }; + recentEntries.push(entry); + if (recentEntries.length > MAX_RECENT_ENTRIES) recentEntries.shift(); + + const prefix = `[${scope}] ${message}`; + const suffix = entry.detail ? `: ${entry.detail}` : ''; + if (level === 'debug') console.debug(prefix + suffix); + else if (level === 'info') console.info(prefix + suffix); + else if (level === 'warn') console.warn(prefix + suffix); + else console.error(prefix + suffix); +} + +export function debugCore(scope: string, message: string, detail?: unknown): void { + emit('debug', scope, message, detail); +} + +export function infoCore(scope: string, message: string, detail?: unknown): void { + emit('info', scope, message, detail); +} + +export function warnCore(scope: string, message: string, detail?: unknown): void { + emit('warn', scope, message, detail); +} + +export function errorCore(scope: string, message: string, detail?: unknown): void { + emit('error', scope, message, detail); +} \ No newline at end of file diff --git a/packages/ai-coach-analytics/src/metric-engine.ts b/packages/ai-coach-analytics/src/metric-engine.ts new file mode 100644 index 0000000..22fc752 --- /dev/null +++ b/packages/ai-coach-analytics/src/metric-engine.ts @@ -0,0 +1,605 @@ +/*--------------------------------------------------------------------------------------------- + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the MIT License. See LICENSE in the project root for license information. + *--------------------------------------------------------------------------------------------*/ + +/** + * Metric engine: parses .metric.md files, evaluates metrics via the DSL, + * runs auto-calibration, validates test fixtures, and resolves rule dependencies. + * + * A .metric.md file defines a reusable data query: + * + * --- + * id: short-messages + * name: Short Messages + * scope: requests + * version: 1 + * --- + * + * # Filter + * messageLength < 30 AND messageLength > 0 + * + * # Metric + * ratio + * + * # Examples + * "{{messageText | truncate:80}}" ({{messageLength}} chars) + */ + +import type { DetectorEmission } from './types/rule-types'; +import { + compileFilter, + compileTrigger, + parseAggregation, + computeAggregation, + evaluateTemplate, +} from './dsl/index'; +import { fillTemplate } from './rule-parser'; + +/* ================================================================== */ +/* Metric Definition (parsed from .metric.md) */ +/* ================================================================== */ + +export interface MetricDefinition { + id: string; + name: string; + scope: 'requests' | 'sessions'; + version: number; + tags: string[]; + /** Raw filter expression (DSL) */ + filterExpr: string; + /** Aggregation expression: ratio, count, avg(field), etc. */ + aggregationExpr: string; + /** Example template for matched items */ + exampleTemplate: string; + /** The full raw markdown source */ + rawSource: string; + /** Where this metric was loaded from */ + source: 'built-in' | 'personal' | 'project'; + sourceFilePath: string; +} + +function isRecord(value: unknown): value is Record { + return typeof value === 'object' && value !== null; +} + +function stringValue(value: unknown): string { + if (typeof value === 'string') return value; + if (typeof value === 'number' || typeof value === 'boolean' || typeof value === 'bigint') { + return `${value}`; + } + return ''; +} + +function numericValue(value: unknown): number | null { + if (typeof value === 'number' && !Number.isNaN(value)) return value; + if (typeof value === 'string') { + const parsed = Number.parseInt(value, 10); + return Number.isNaN(parsed) ? null : parsed; + } + return null; +} + +function stringListValue(value: unknown): string[] { + if (Array.isArray(value)) return value.map(item => stringValue(item)).filter(Boolean); + const single = stringValue(value); + return single ? single.split(/[,\s]+/).filter(Boolean) : []; +} + +function firstRowText(row: Record, ...keys: string[]): string { + for (const key of keys) { + const text = stringValue(row[key]); + if (text) return text; + } + return ''; +} + +/* ================================================================== */ +/* Metric Parser */ +/* ================================================================== */ + +export function parseMetric(markdown: string): MetricDefinition | null { + const fmMatch = markdown.match(/^---\s*\n([\s\S]*?)\n---\s*\n/); + if (!fmMatch) return null; + + const fm = parseYamlLike(fmMatch[1]); + const id = stringValue(fm.id); + const name = stringValue(fm.name); + if (!id || !name) return null; + + const body = markdown.substring(fmMatch[0].length); + const filterExpr = extractSection(body, 'Filter') || ''; + const aggregationExpr = extractSection(body, 'Metric') || 'ratio'; + const exampleTemplate = extractSection(body, 'Examples') || ''; + + return { + id, + name, + scope: fm.scope === 'sessions' ? 'sessions' : 'requests', + version: numericValue(fm.version) || 1, + tags: stringListValue(fm.tags), + filterExpr, + aggregationExpr, + exampleTemplate, + rawSource: markdown, + source: 'built-in', + sourceFilePath: '', + }; +} + +/* ================================================================== */ +/* Enhanced Rule Extensions (parsed from rule .md # sections) */ +/* ================================================================== */ + +export interface RuleExtensions { + /** Inline filter expression (from # Filter section) */ + filterExpr: string; + /** Trigger expression (from # Trigger section) */ + triggerExpr: string; + /** Referenced metric ID (from frontmatter metric: field) */ + metricRef: string; + /** Rule dependency IDs (from frontmatter requires: field) */ + requires: string[]; + /** Test fixtures */ + testCases: TestCase[]; +} + +export interface TestCase { + input: Record; + expect: 'flagged' | 'clean'; +} + +/** + * Extract extended sections from a rule markdown body + frontmatter. + * Called after the standard parseRule() to augment with DSL-powered sections. + */ +export function parseRuleExtensions(markdown: string): RuleExtensions { + const fmMatch = markdown.match(/^---\s*\n([\s\S]*?)\n---\s*\n/); + const fm = fmMatch ? parseYamlLike(fmMatch[1]) : {}; + const body = fmMatch ? markdown.substring(fmMatch[0].length) : markdown; + + const filterExpr = extractSection(body, 'Filter') || ''; + const triggerExpr = extractSection(body, 'Trigger') || ''; + const metricRef = stringValue(fm.metric); + const requires = stringListValue(fm.requires); + const testCases = parseTestCases(body); + + return { filterExpr, triggerExpr, metricRef, requires, testCases }; +} + +/* ================================================================== */ +/* Metric Evaluation */ +/* ================================================================== */ + +/** + * Evaluate a metric (from .metric.md or inline rule # Filter) against data. + * Returns a DetectorEmission. + */ +export function evaluateMetric( + filterExpr: string, + aggregationExpr: string, + exampleTemplate: string, + rows: Record[], + thresholds?: Record, +): DetectorEmission { + // Resolve threshold placeholders in the filter expression + let resolvedFilter = filterExpr; + if (thresholds) { + resolvedFilter = fillTemplate(filterExpr, { thresholds } as unknown as Record); + } + + const filterFn = compileFilter(resolvedFilter); + const matched = rows.filter(filterFn); + const agg = parseAggregation(aggregationExpr); + const value = computeAggregation(agg, matched, rows); + + const count = agg.type === 'count' || agg.type === 'ratio' ? matched.length : value; + const total = rows.length; + const ratio = total > 0 ? matched.length / total : 0; + + // Generate examples + const examples: string[] = []; + const sampleRows = matched.slice(0, 3); + for (const row of sampleRows) { + if (exampleTemplate) { + examples.push(evaluateTemplate(exampleTemplate, row)); + } else { + // Default example: first 80 chars of messageText + const msg = firstRowText(row, 'messageText', 'workspaceName', 'sessionId'); + examples.push(msg.substring(0, 80)); + } + } + + return { + count, + total, + ratio, + examples, + extra: { + matchedCount: matched.length, + aggregationType: agg.type, + ...(agg.field ? { aggregationField: agg.field } : {}), + ...(agg.type !== 'count' && agg.type !== 'ratio' ? { aggregationValue: value } : {}), + }, + }; +} + +/** + * Evaluate a trigger expression against an emission. + * Returns true if the rule should fire. + */ +export function evaluateTrigger( + triggerExpr: string, + emission: DetectorEmission, + thresholds?: Record, +): boolean { + if (!triggerExpr.trim()) { + // Default trigger: any matches = triggered + return emission.count > 0; + } + + let resolved = triggerExpr; + if (thresholds) { + resolved = fillTemplate(triggerExpr, { thresholds } as unknown as Record); + } + + const triggerFn = compileTrigger(resolved); + return triggerFn({ + count: emission.count, + total: emission.total, + ratio: emission.ratio, + extra: emission.extra, + }); +} + +/* ================================================================== */ +/* Auto-Calibration */ +/* ================================================================== */ + +export interface CalibrationResult { + datasetSize: number; + scope: string; + flaggedCount: number; + flaggedPct: number; + /** Field distribution stats (for the metric's filter field) */ + distribution: { + p25: number; + p50: number; + p75: number; + min: number; + max: number; + } | null; + triggered: boolean; + lastRun: string; +} + +/** + * Run auto-calibration: evaluate the metric against real data and produce + * distribution stats. This is injected back into the .md file as a comment. + */ +export function calibrate( + filterExpr: string, + aggregationExpr: string, + triggerExpr: string, + rows: Record[], + thresholds?: Record, +): CalibrationResult { + const emission = evaluateMetric(filterExpr, aggregationExpr, '', rows, thresholds); + const triggered = evaluateTrigger(triggerExpr, emission, thresholds); + + // Try to extract the primary field from the filter to compute distribution + let distribution: CalibrationResult['distribution'] = null; + const fieldMatch = filterExpr.match(/^(\w[\w.]*)\s*[<>=!]/); + if (fieldMatch) { + const field = fieldMatch[1]; + const values = rows.map(r => { + const parts = field.split('.'); + let v: unknown = r; + for (const p of parts) { + if (v == null) return Number.NaN; + if (typeof v === 'object' && !Array.isArray(v)) v = (v as Record)[p]; + else if (Array.isArray(v) && p === 'length') return v.length; + else return Number.NaN; + } + return typeof v === 'number' ? v : Number.NaN; + }).filter(n => !Number.isNaN(n)).sort((a, b) => a - b); + + if (values.length > 0) { + distribution = { + min: values[0], + max: values[values.length - 1], + p25: values[Math.floor(values.length * 0.25)], + p50: values[Math.floor(values.length * 0.50)], + p75: values[Math.floor(values.length * 0.75)], + }; + } + } + + return { + datasetSize: rows.length, + scope: aggregationExpr.includes('session') ? 'sessions' : 'requests', + flaggedCount: emission.count, + flaggedPct: rows.length > 0 ? Math.round((emission.count / rows.length) * 1000) / 10 : 0, + distribution, + triggered, + lastRun: new Date().toISOString().split('T')[0], + }; +} + +/** + * Serialize a CalibrationResult as a markdown comment block. + */ +export function serializeCalibration(cal: CalibrationResult): string { + const lines = [''); + return lines.join('\n'); +} + +/** + * Strip any existing calibration comment from markdown source. + */ +export function stripCalibration(markdown: string): string { + return markdown.replace(/\n?\s*$/, '').trimEnd(); +} + +/* ================================================================== */ +/* Test Fixture Runner */ +/* ================================================================== */ + +export interface TestResult { + index: number; + expected: 'flagged' | 'clean'; + actual: 'flagged' | 'clean'; + passed: boolean; + emission?: DetectorEmission; +} + +/** + * Run test cases for a rule/metric. + * Each test case provides a partial input object and the expected outcome. + * The engine evaluates the filter + trigger against a synthetic dataset + * containing just the test input. + */ +export function runTestCases( + filterExpr: string, + aggregationExpr: string, + triggerExpr: string, + testCases: TestCase[], + thresholds?: Record, +): TestResult[] { + return testCases.map((tc, index) => { + // Create a minimal dataset with just this one row + const row = { ...defaultRow(), ...tc.input }; + const rows = [row]; + + const emission = evaluateMetric(filterExpr, aggregationExpr, '', rows, thresholds); + const triggered = evaluateTrigger(triggerExpr, emission, thresholds); + const actual = triggered ? 'flagged' : 'clean'; + + return { + index, + expected: tc.expect, + actual, + passed: tc.expect === actual, + emission, + }; + }); +} + +/** Default row values for test fixtures (zero/empty for all fields). */ +function defaultRow(): Record { + return { + requestId: 'test-req', + timestamp: Date.now(), + messageText: '', + responseText: '', + isCanceled: false, + agentName: '', + agentMode: 'chat', + modelId: 'gpt-4.1', + toolsUsed: [], + editedFiles: [], + referencedFiles: [], + slashCommand: '', + variableKinds: {}, + customInstructions: [], + skillsUsed: [], + firstProgress: null, + totalElapsed: null, + messageLength: 0, + responseLength: 0, + userCode: [], + aiCode: [], + toolConfirmations: [], + promptTokens: null, + completionTokens: null, + cacheReadTokens: null, + cacheWriteTokens: null, + workType: '', + // Session fields + sessionId: 'test-session', + workspaceId: 'test-ws', + workspaceName: 'Test Workspace', + location: '/tmp/test', + harness: 'VS Code', + creationDate: Date.now(), + lastMessageDate: Date.now(), + requestCount: 1, + requests: [], + }; +} + +/* ================================================================== */ +/* Dependency Resolution */ +/* ================================================================== */ + +export interface DependencyGraph { + /** Topological order of rule IDs (dependencies before dependents) */ + order: string[]; + /** Rules with unresolvable dependencies */ + errors: { ruleId: string; missing: string[] }[]; +} + +/** + * Resolve a dependency graph for rules that have `requires:` fields. + * Returns a topological ordering so dependent rules run after their prerequisites. + */ +export function resolveDependencies( + ruleRequires: Map, +): DependencyGraph { + const allIds = new Set(ruleRequires.keys()); + const visited = new Set(); + const visiting = new Set(); + const order: string[] = []; + const errors: DependencyGraph['errors'] = []; + + function visit(id: string): void { + if (visited.has(id)) return; + if (visiting.has(id)) return; // cycle - skip + + visiting.add(id); + const deps = ruleRequires.get(id) || []; + const missing = deps.filter(d => !allIds.has(d)); + if (missing.length > 0) { + errors.push({ ruleId: id, missing }); + } + for (const dep of deps) { + if (allIds.has(dep)) visit(dep); + } + visiting.delete(id); + visited.add(id); + order.push(id); + } + + for (const id of allIds) { + visit(id); + } + + return { order, errors }; +} + +/* ================================================================== */ +/* Metric Registry (in-memory store) */ +/* ================================================================== */ + +const METRIC_STORE = new Map(); + +export function registerMetric(metric: MetricDefinition): void { + METRIC_STORE.set(metric.id, metric); +} + +export function getAllMetrics(): MetricDefinition[] { + return [...METRIC_STORE.values()]; +} + +export function clearMetrics(): void { + METRIC_STORE.clear(); +} + +/* ================================================================== */ +/* Internal helpers */ +/* ================================================================== */ + +function extractSection(body: string, heading: string): string | null { + const re = new RegExp(`^#\\s+${heading}\\s*$`, 'mi'); + const match = re.exec(body); + if (!match) return null; + const start = match.index + match[0].length; + const nextHeading = body.indexOf('\n#', start); + const section = nextHeading >= 0 ? body.substring(start, nextHeading) : body.substring(start); + return section.trim() || null; +} + +function parseYamlLike(text: string): Record { + const result: Record = {}; + const lines = text.split('\n'); + let currentKey = ''; + let indent = 0; + const arrayBuffer: string[] = []; + + for (const line of lines) { + const trimmed = line.trim(); + if (!trimmed || trimmed.startsWith('#')) continue; + + // Array item under a key + if (trimmed.startsWith('- ') && currentKey && indent > 0) { + arrayBuffer.push(trimmed.substring(2).trim()); + continue; + } + + // Flush array buffer + if (arrayBuffer.length > 0 && currentKey) { + result[currentKey] = arrayBuffer.slice(); + arrayBuffer.length = 0; + } + + const colonIdx = trimmed.indexOf(':'); + if (colonIdx < 0) continue; + + const key = trimmed.substring(0, colonIdx).trim(); + const val = trimmed.substring(colonIdx + 1).trim(); + currentKey = key; + indent = line.length - line.trimStart().length; + + if (!val) continue; // Sub-object or array follows + + // Inline array: [a, b, c] + if (val.startsWith('[') && val.endsWith(']')) { + result[key] = val.slice(1, -1).split(',').map(s => { + const t = s.trim(); + const n = Number.parseFloat(t); + return !Number.isNaN(n) && /^[\d.-]+$/.test(t) ? n : t; + }); + } else if (val === 'true' || val === 'false') { + result[key] = val === 'true'; + } else { + const num = Number.parseFloat(val); + result[key] = !Number.isNaN(num) && /^[\d.-]+$/.test(val) ? num : val; + } + } + + // Flush remaining array + if (arrayBuffer.length > 0 && currentKey) { + result[currentKey] = arrayBuffer.slice(); + } + + return result; +} + +function parseTestCases(body: string): TestCase[] { + const section = extractSection(body, 'Test Cases'); + if (!section) return []; + + const cases: TestCase[] = []; + + // Support YAML-like list format: + // - input: { field: value } + // expect: flagged + const blocks = ('\n' + section).split(/\n\s*-\s+input:/); + for (let i = 1; i < blocks.length; i++) { + const block = blocks[i]; + try { + // Extract input JSON + const inputMatch = block.match(/^\s*(\{[\s\S]*?\})/m); + const expectMatch = block.match(/expect:\s*(flagged|clean)/i); + if (inputMatch && expectMatch) { + const input = JSON.parse(inputMatch[1]) as unknown; + if (!isRecord(input)) continue; + const expect = expectMatch[1].toLowerCase() as 'flagged' | 'clean'; + cases.push({ input, expect }); + } + } catch { + // Skip unparseable test cases + } + } + + return cases; +} diff --git a/packages/ai-coach-analytics/src/metrics/agentic-no-tools.metric.md b/packages/ai-coach-analytics/src/metrics/agentic-no-tools.metric.md new file mode 100644 index 0000000..1bfbd80 --- /dev/null +++ b/packages/ai-coach-analytics/src/metrics/agentic-no-tools.metric.md @@ -0,0 +1,16 @@ +--- +id: agentic-no-tools +name: Agentic Without Tools +scope: requests +version: 1 +tags: [tool, agent] +--- + +# Filter +agentMode == "agent" AND toolsUsed.length == 0 + +# Metric +ratio + +# Examples +"{{messageText | truncate:80}}" ({{agentMode}}, {{modelId}}) diff --git a/packages/ai-coach-analytics/src/metrics/canceled-requests.metric.md b/packages/ai-coach-analytics/src/metrics/canceled-requests.metric.md new file mode 100644 index 0000000..a74f552 --- /dev/null +++ b/packages/ai-coach-analytics/src/metrics/canceled-requests.metric.md @@ -0,0 +1,16 @@ +--- +id: canceled-requests +name: Canceled Requests +scope: requests +version: 1 +tags: [session, cancel] +--- + +# Filter +isCanceled == true + +# Metric +ratio + +# Examples +"{{messageText | truncate:80}}" ({{modelId}}) diff --git a/packages/ai-coach-analytics/src/metrics/capslock-messages.metric.md b/packages/ai-coach-analytics/src/metrics/capslock-messages.metric.md new file mode 100644 index 0000000..e4c32c5 --- /dev/null +++ b/packages/ai-coach-analytics/src/metrics/capslock-messages.metric.md @@ -0,0 +1,16 @@ +--- +id: capslock-messages +name: All-Caps Messages +scope: requests +version: 1 +tags: [prompt, frustration] +--- + +# Filter +matches(messageText, "/^[A-Z\\s!?.,]{20,}/") + +# Metric +count + +# Examples +"{{messageText | truncate:80}}" diff --git a/packages/ai-coach-analytics/src/metrics/late-night-requests.metric.md b/packages/ai-coach-analytics/src/metrics/late-night-requests.metric.md new file mode 100644 index 0000000..82b0968 --- /dev/null +++ b/packages/ai-coach-analytics/src/metrics/late-night-requests.metric.md @@ -0,0 +1,16 @@ +--- +id: late-night-requests +name: Late Night Requests +scope: requests +version: 1 +tags: [wellbeing, time] +--- + +# Filter +hour(timestamp) >= 22 OR hour(timestamp) < 6 + +# Metric +ratio + +# Examples +{{messageText | truncate:60}} at {{hour(timestamp)}}:00 diff --git a/packages/ai-coach-analytics/src/metrics/mega-sessions.metric.md b/packages/ai-coach-analytics/src/metrics/mega-sessions.metric.md new file mode 100644 index 0000000..42f0c1c --- /dev/null +++ b/packages/ai-coach-analytics/src/metrics/mega-sessions.metric.md @@ -0,0 +1,16 @@ +--- +id: mega-sessions +name: Mega Sessions +scope: sessions +version: 1 +tags: [session, size] +--- + +# Filter +requestCount > 40 + +# Metric +count + +# Examples +{{workspaceName}} ({{requestCount}} requests) diff --git a/packages/ai-coach-analytics/src/metrics/no-custom-instructions.metric.md b/packages/ai-coach-analytics/src/metrics/no-custom-instructions.metric.md new file mode 100644 index 0000000..fc11da9 --- /dev/null +++ b/packages/ai-coach-analytics/src/metrics/no-custom-instructions.metric.md @@ -0,0 +1,16 @@ +--- +id: no-custom-instructions +name: No Custom Instructions +scope: requests +version: 1 +tags: [tool, instructions] +--- + +# Filter +customInstructions.length == 0 + +# Metric +ratio + +# Examples +"{{messageText | truncate:80}}" ({{agentMode}}) diff --git a/packages/ai-coach-analytics/src/metrics/no-file-refs.metric.md b/packages/ai-coach-analytics/src/metrics/no-file-refs.metric.md new file mode 100644 index 0000000..4980b62 --- /dev/null +++ b/packages/ai-coach-analytics/src/metrics/no-file-refs.metric.md @@ -0,0 +1,16 @@ +--- +id: no-file-refs +name: No File References +scope: requests +version: 1 +tags: [context, files] +--- + +# Filter +referencedFiles.length == 0 AND editedFiles.length == 0 + +# Metric +ratio + +# Examples +"{{messageText | truncate:80}}" diff --git a/packages/ai-coach-analytics/src/metrics/short-messages.metric.md b/packages/ai-coach-analytics/src/metrics/short-messages.metric.md new file mode 100644 index 0000000..82acfb4 --- /dev/null +++ b/packages/ai-coach-analytics/src/metrics/short-messages.metric.md @@ -0,0 +1,16 @@ +--- +id: short-messages +name: Short Messages +scope: requests +version: 1 +tags: [prompt, quality, length] +--- + +# Filter +messageLength < 30 AND messageLength > 0 + +# Metric +ratio + +# Examples +"{{messageText | truncate:80}}" ({{messageLength}} chars) diff --git a/packages/ai-coach-analytics/src/metrics/slow-responses.metric.md b/packages/ai-coach-analytics/src/metrics/slow-responses.metric.md new file mode 100644 index 0000000..630a668 --- /dev/null +++ b/packages/ai-coach-analytics/src/metrics/slow-responses.metric.md @@ -0,0 +1,16 @@ +--- +id: slow-responses +name: Slow Responses +scope: requests +version: 1 +tags: [performance, latency] +--- + +# Filter +totalElapsed > 30000 + +# Metric +count + +# Examples +"{{messageText | truncate:60}}" took {{totalElapsed}}ms ({{modelId}}) diff --git a/packages/ai-coach-analytics/src/metrics/weekend-requests.metric.md b/packages/ai-coach-analytics/src/metrics/weekend-requests.metric.md new file mode 100644 index 0000000..e6d2a8c --- /dev/null +++ b/packages/ai-coach-analytics/src/metrics/weekend-requests.metric.md @@ -0,0 +1,16 @@ +--- +id: weekend-requests +name: Weekend Requests +scope: requests +version: 1 +tags: [wellbeing, time] +--- + +# Filter +dayOfWeek(timestamp) == 0 OR dayOfWeek(timestamp) == 6 + +# Metric +ratio + +# Examples +{{messageText | truncate:60}} ({{dayOfWeek(timestamp)}}) diff --git a/packages/ai-coach-analytics/src/parser-claude.ts b/packages/ai-coach-analytics/src/parser-claude.ts new file mode 100644 index 0000000..5e12059 --- /dev/null +++ b/packages/ai-coach-analytics/src/parser-claude.ts @@ -0,0 +1,682 @@ +/*--------------------------------------------------------------------------------------------- + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the MIT License. See LICENSE in the project root for license information. + *--------------------------------------------------------------------------------------------*/ + +/* Claude session parser + * + * Data layout (macOS): + * ~/.claude/projects//.jsonl + * + * Each .jsonl file is a session. Lines have { type: 'user'|'assistant'|'queue-operation'|'last-prompt', ... } + * + * user lines: message.content[].text, timestamp, cwd, sessionId, version, gitBranch + * assistant lines: message.content[] with type 'thinking'|'text'|'tool_use', + * message.model, message.usage, timestamp + */ + +import * as fs from 'fs'; +import * as path from 'path'; +import { Session, SessionRequest } from './types'; +import { assertTrustedPath, readFileSafe, createRequest, createSession, detectDevcontainerFromRequests, extractSkillNameFromPath } from './parser-shared'; +import { extractReasoningEffortFromModelId } from './helpers'; +import { warnCore } from './log'; + +interface ClaudeContentBlock { + type: string; + text?: string; + name?: string; + input?: Record; +} + +interface ClaudeMessage { + role?: string; + model?: string; + content?: ClaudeContentBlock[] | string; + usage?: Record; +} + +interface ClaudeLine { + type: string; + uuid?: string; + parentUuid?: string; + sessionId?: string; + timestamp?: string; + cwd?: string; + version?: string; + gitBranch?: string; + /** Set from CLAUDE_CODE_ENTRYPOINT env var at launch time. Known values: + * cli, sdk-ts, sdk-py, sdk-cli, mcp, claude-code-github-action, + * claude-desktop, local-agent. Used to classify the session as + * interactive (cli, claude-desktop) vs programmatic (everything else). */ + entrypoint?: string; + /** True on subagent lines (`/subagents/agent-*.jsonl` files). + * Carried through so callers can roll subagent requests up into the + * parent session rather than treating them as standalone sessions. */ + isSidechain?: boolean; + /** Subagent identifier on `subagents/` files. Present alongside + * `isSidechain: true`. Not used for billing math; surfaced for diagnostics. */ + agentId?: string; + message?: ClaudeMessage; +} + +/** Allow-list of `entrypoint` values that indicate the user launched Claude + * Code interactively. Anything outside this set (sdk-ts, sdk-py, mcp, + * claude-code-github-action, local-agent, unknown, missing) is treated as + * programmatic — i.e. spawned by another tool. Defaulting unknown values to + * programmatic is the safer choice: it keeps standalone Claude usage + * conservative (under-attributed rather than over-attributed). */ +const INTERACTIVE_ENTRYPOINTS: ReadonlySet = new Set(['cli', 'claude-desktop']); + +function classifyLauncher(entrypoint: string | undefined): 'interactive' | 'programmatic' { + return entrypoint && INTERACTIVE_ENTRYPOINTS.has(entrypoint) ? 'interactive' : 'programmatic'; +} + +function harnessForLauncher(_launcherKind: 'interactive' | 'programmatic'): string { + return 'Claude'; +} + +interface ClaudeAssistantData { + nextIndex: number; + lastTs: number | null; + assistantTexts: string[]; + toolsUsed: string[]; + editedFiles: string[]; + referencedFiles: string[]; + skillsUsed: string[]; + model: string; + totalInputTokens: number; + totalOutputTokens: number; + totalCacheReadTokens: number; + totalCacheWriteTokens: number; + assistantCount: number; +} + +const CLAUDE_WRITE_TOOLS = new Set(['Write', 'Edit', 'MultiEditTool']); +const CLAUDE_READ_FILE_TOOLS = new Set(['Read', 'View']); +const CLAUDE_READ_PATH_TOOLS = new Set(['Glob', 'LS', 'Find']); + +function isRecord(value: unknown): value is Record { + return typeof value === 'object' && value !== null && !Array.isArray(value); +} + +function isClaudeContentBlock(value: unknown): value is ClaudeContentBlock { + if (!isRecord(value) || typeof value.type !== 'string') return false; + if (value.text !== undefined && typeof value.text !== 'string') return false; + if (value.name !== undefined && typeof value.name !== 'string') return false; + if (value.input !== undefined && value.input !== null && !isRecord(value.input)) return false; + return true; +} + +function isClaudeMessage(value: unknown): value is ClaudeMessage { + if (!isRecord(value)) return false; + if (value.role !== undefined && typeof value.role !== 'string') return false; + if (value.model !== undefined && typeof value.model !== 'string') return false; + if (value.content !== undefined) { + const content = value.content; + if (typeof content !== 'string' && (!Array.isArray(content) || !content.every(isClaudeContentBlock))) { + return false; + } + } + if (value.usage !== undefined && !isRecord(value.usage)) return false; + return true; +} + +function isClaudeLine(value: unknown): value is ClaudeLine { + if (!isRecord(value) || typeof value.type !== 'string') return false; + if (value.uuid !== undefined && typeof value.uuid !== 'string' && typeof value.uuid !== 'number') return false; + if (value.parentUuid !== undefined && value.parentUuid !== null && typeof value.parentUuid !== 'string' && typeof value.parentUuid !== 'number') return false; + if (value.sessionId !== undefined && typeof value.sessionId !== 'string') return false; + if (value.timestamp !== undefined && typeof value.timestamp !== 'string' && typeof value.timestamp !== 'number') return false; + if (value.cwd !== undefined && typeof value.cwd !== 'string') return false; + if (value.version !== undefined && typeof value.version !== 'string') return false; + if (value.gitBranch !== undefined && typeof value.gitBranch !== 'string') return false; + if (value.entrypoint !== undefined && typeof value.entrypoint !== 'string') return false; + if (value.isSidechain !== undefined && typeof value.isSidechain !== 'boolean') return false; + if (value.agentId !== undefined && typeof value.agentId !== 'string') return false; + if (value.message !== undefined && !isClaudeMessage(value.message)) return false; + return true; +} + +function parseClaudeLine(rawLine: string): ClaudeLine | null { + try { + const parsed: unknown = JSON.parse(rawLine); + if (!isClaudeLine(parsed)) return null; + const record = parsed as unknown as Record; + if (typeof record.uuid === 'number') record.uuid = String(record.uuid); + if (typeof record.parentUuid === 'number') record.parentUuid = String(record.parentUuid); + if (typeof record.timestamp === 'number') record.timestamp = new Date(record.timestamp).toISOString(); + return parsed; + } catch { + return null; + } +} + +function parseClaudeLines(raw: string): ClaudeLine[] { + const lines: ClaudeLine[] = []; + for (const rawLine of raw.split('\n')) { + if (!rawLine.trim()) continue; + const parsed = parseClaudeLine(rawLine); + if (parsed) lines.push(parsed); + } + return lines; +} + +function toContentArray(content: ClaudeContentBlock[] | string | undefined): ClaudeContentBlock[] { + if (Array.isArray(content)) return content; + if (typeof content === 'string') return [{ type: 'text', text: content }]; + return []; +} + +function userHasText(line: ClaudeLine): boolean { + const content = line.message?.content; + let text: string; + if (typeof content === 'string') { + text = content.trim(); + } else if (Array.isArray(content)) { + const parts: string[] = []; + for (const block of content) { + if (block.type === 'text' && block.text) parts.push(block.text); + } + text = parts.join('\n').trim(); + } else { + return false; + } + if (text.length === 0) return false; + // Claude wraps slash-command input/output in / tags + if (/^<(local-)?command-/.test(text)) return false; + // Interrupt markers are auto-injected when the user cancels a tool call + if (text.startsWith('[Request interrupted')) return false; + return true; +} + +function getTimestamp(timestamp?: string): number | null { + return timestamp ? new Date(timestamp).getTime() : null; +} + +function getNumberField(record: Record | undefined, key: string): number { + const value = record?.[key]; + return typeof value === 'number' ? value : 0; +} + +function getInputPath(input: Record | undefined, key: string): string | null { + const value = input?.[key]; + return typeof value === 'string' ? value : null; +} + +function getClaudeUserText(line: ClaudeLine): string { + return toContentArray(line.message?.content) + .filter(block => block.type === 'text') + .map(block => block.text || '') + .join('\n'); +} + +function countClaudeImages(line: ClaudeLine): number { + return toContentArray(line.message?.content) + .filter(block => block.type === 'image').length; +} + +function applyClaudeToolBlock( + block: ClaudeContentBlock, + data: Pick, +): void { + if (block.type !== 'tool_use' || !block.name) return; + + data.toolsUsed.push(block.name); + + // Claude Code's Skill tool: { name: 'Skill', input: { skill: '', args: '...' } } + if (block.name === 'Skill') { + const skillName = getInputPath(block.input, 'skill')?.trim(); + if (skillName && !skillName.includes('ai_toolkit')) data.skillsUsed.push(skillName); + return; + } + if (CLAUDE_WRITE_TOOLS.has(block.name)) { + const filePath = getInputPath(block.input, 'file_path'); + if (filePath) data.editedFiles.push(filePath); + return; + } + + if (CLAUDE_READ_FILE_TOOLS.has(block.name)) { + const filePath = getInputPath(block.input, 'file_path'); + if (filePath) data.referencedFiles.push(filePath); + return; + } + + if (CLAUDE_READ_PATH_TOOLS.has(block.name)) { + const targetPath = getInputPath(block.input, 'path'); + if (targetPath) data.referencedFiles.push(targetPath); + } +} + +function collectClaudeAssistantData(lines: ClaudeLine[], startIndex: number, lastTs: number | null): ClaudeAssistantData { + const data: ClaudeAssistantData = { + nextIndex: startIndex, + lastTs, + assistantTexts: [], + toolsUsed: [], + editedFiles: [], + referencedFiles: [], + skillsUsed: [], + model: '', + totalInputTokens: 0, + totalOutputTokens: 0, + totalCacheReadTokens: 0, + totalCacheWriteTokens: 0, + assistantCount: 0, + }; + + let i = startIndex; + while (i < lines.length) { + const next = lines[i]; + if (next.type === 'user' && userHasText(next)) break; + if (next.type === 'assistant') { + data.assistantCount++; + const assistantTs = getTimestamp(next.timestamp); + if (assistantTs && (!data.lastTs || assistantTs > data.lastTs)) data.lastTs = assistantTs; + if (!data.model && next.message?.model) data.model = next.message.model; + + const usage = next.message?.usage; + const cacheRead = getNumberField(usage, 'cache_read_input_tokens'); + const cacheWrite = getNumberField(usage, 'cache_creation_input_tokens'); + data.totalInputTokens += getNumberField(usage, 'input_tokens') + cacheRead + cacheWrite; + data.totalOutputTokens += getNumberField(usage, 'output_tokens'); + data.totalCacheReadTokens += cacheRead; + data.totalCacheWriteTokens += cacheWrite; + + for (const block of toContentArray(next.message?.content)) { + if (block.type === 'text' && block.text) { + data.assistantTexts.push(block.text); + continue; + } + applyClaudeToolBlock(block, data); + // Extract code content from write tools so extractCodeBlocks() can count LoC. + // Claude Write uses `content`, Edit uses `new_str`. + if (block.type === 'tool_use' && block.name && CLAUDE_WRITE_TOOLS.has(block.name) && block.input) { + const filePath = getInputPath(block.input, 'file_path'); + const code = typeof block.input.content === 'string' ? block.input.content + : typeof block.input.new_str === 'string' ? block.input.new_str + : null; + if (code && filePath) { + const ext = filePath.split('.').pop() || 'unknown'; + data.assistantTexts.push(`\`\`\`${ext}\n${code}\n\`\`\``); + } + } + } + } + i++; + } + + data.nextIndex = i; + return data; +} + +export function findClaudeDirs(): string[] { + const home = process.env.HOME || process.env.USERPROFILE || ''; + const dirs: string[] = []; + const projectsDir = path.join(home, '.claude', 'projects'); + if (fs.existsSync(projectsDir)) dirs.push(projectsDir); + return dirs; +} + +/** Encode a single filesystem component the way Claude Code does: + * replace whitespace with hyphens. Path separators never appear in a single + * component name, and colons are not handled here because Windows component + * names cannot contain them. */ +function encodeComponentForMatch(name: string): string { + return name.replace(/\s/g, '-'); +} + +/** + * Resolve an encoded Claude project directory name back to the real folder name. + * + * Claude encodes workspace paths by replacing `/`, `\`, `:`, and whitespace + * with `-`. For example `c--dev-AI-Engineering-Coach` represents + * `C:\dev\AI-Engineering-Coach`. Because the encoding is lossy (hyphens, spaces, + * and path separators all become `-`) we resolve the name by listing each + * directory level and matching entries against the remaining encoded string. + * + * @param encoded The encoded project directory name. + * @param _projectsDir The `.claude/projects` directory containing this project. + */ +function projectNameFromEncoded(encoded: string, _projectsDir: string): string { + const segments = encoded.split('-'); + let root: string; + let startIdx: number; + + // Windows drive: encoded starts with "--..." → "X:\\" + if (segments.length >= 2 && /^[a-zA-Z]$/.test(segments[0]) && segments[1] === '') { + root = `${segments[0]}:\\`; + startIdx = 2; + // Unix-style absolute path: encoded starts with "-..." → "/" + } else if (segments[0] === '') { + root = '/'; + startIdx = 1; + } else { + return encoded; // unexpected format + } + + // Join remaining segments back into the encoded tail so we can match + // against directory entries whose names may contain spaces or hyphens. + const remaining = segments.slice(startIdx).join('-'); + let resolved = root; + let offset = 0; + + while (offset < remaining.length) { + let dirEntries: { name: string; encoded: string }[]; + try { + dirEntries = fs.readdirSync(resolved, { withFileTypes: true }) + .filter(e => e.isDirectory() || e.isSymbolicLink()) + .map(e => ({ name: e.name, encoded: encodeComponentForMatch(e.name) })) + // Longest encoded form first — greedy match avoids splitting names + // that contain hyphens (e.g. `AI-Engineering-Coach`). + .sort((a, b) => b.encoded.length - a.encoded.length); + } catch { + break; + } + + const rest = remaining.slice(offset); + let found = false; + for (const entry of dirEntries) { + if (rest === entry.encoded) { + // Exact match — last component + resolved = path.join(resolved, entry.name); + offset = remaining.length; + found = true; + break; + } + if (rest.startsWith(entry.encoded + '-')) { + resolved = path.join(resolved, entry.name); + offset += entry.encoded.length + 1; + found = true; + break; + } + } + + if (!found) { + // Can't resolve further — take the rest as one segment + resolved = path.join(resolved, rest); + break; + } + } + + return path.basename(resolved); +} + +function parseClaudeProjectSessions( + projectsDir: string, + dirName: string, +): { sessions: Session[]; workspaceId: string; workspaceName: string } | null { + const projPath = path.join(projectsDir, dirName); + const workspaceId = `claude-${dirName}`; + const workspaceName = projectNameFromEncoded(dirName, projectsDir); + + let entries: fs.Dirent[]; + try { + entries = fs.readdirSync(projPath, { withFileTypes: true }); + } catch { + return null; + } + + // Pass 1: parse parent sessions from top-level .jsonl files. Index them by + // sessionId so subagent files (in `/subagents/agent-*.jsonl`) + // can be merged into the right parent in pass 2. + const sessionsById = new Map(); + const sessions: Session[] = []; + for (const entry of entries) { + if (!entry.isFile() || !entry.name.endsWith('.jsonl')) continue; + const session = parseClaudeSessionFile(path.join(projPath, entry.name), workspaceId, workspaceName); + if (!session) continue; + sessions.push(session); + sessionsById.set(session.sessionId, session); + } + + // Pass 2: walk `/subagents/agent-*.jsonl` directories. Each + // subagent file is a fan-out of work the parent session orchestrated; we + // merge its requests into the parent rather than emitting it as its own + // session. Orphans (subagent dir whose parent session is missing) become + // standalone programmatic sessions — they're still GHCP-spawned work. + for (const entry of entries) { + if (!entry.isDirectory()) continue; + const subagentDir = path.join(projPath, entry.name, 'subagents'); + let subagentEntries: fs.Dirent[]; + try { + subagentEntries = fs.readdirSync(subagentDir, { withFileTypes: true }); + } catch { + continue; // no subagents/ folder under this session — fine + } + + const parent = sessionsById.get(entry.name); + const orphanRequests: SessionRequest[] = []; + let orphanFirstTs: number | null = null; + let orphanLastTs: number | null = null; + let orphanEntrypoint: string | undefined; + + for (const subEntry of subagentEntries) { + if (!subEntry.isFile() || !subEntry.name.endsWith('.jsonl')) continue; + const subSession = parseClaudeSessionFile( + path.join(subagentDir, subEntry.name), workspaceId, workspaceName, + ); + if (!subSession) continue; + + if (parent) { + // Merge subagent requests into parent. Sorting happens after all + // subagents are collected, below. + for (const r of subSession.requests) parent.requests.push(r); + // Extend parent's last-message timestamp if the subagent ran longer. + if (subSession.lastMessageDate && + (!parent.lastMessageDate || subSession.lastMessageDate > parent.lastMessageDate)) { + parent.lastMessageDate = subSession.lastMessageDate; + } + } else { + for (const r of subSession.requests) orphanRequests.push(r); + if (subSession.creationDate && + (!orphanFirstTs || subSession.creationDate < orphanFirstTs)) { + orphanFirstTs = subSession.creationDate; + } + if (subSession.lastMessageDate && + (!orphanLastTs || subSession.lastMessageDate > orphanLastTs)) { + orphanLastTs = subSession.lastMessageDate; + } + if (!orphanEntrypoint && subSession.entrypoint) orphanEntrypoint = subSession.entrypoint; + } + } + + if (parent) continue; + + if (orphanRequests.length > 0) { + warnCore('parser-claude', `subagent dir without parent session: ${entry.name}`, { + requestCount: orphanRequests.length, + }); + orphanRequests.sort((a, b) => (a.timestamp ?? 0) - (b.timestamp ?? 0)); + const launcherKind: 'interactive' | 'programmatic' = 'programmatic'; + const orphan = createSession({ + sessionId: entry.name, + workspaceId, + workspaceName, + location: 'terminal', + harness: harnessForLauncher(launcherKind), + creationDate: orphanFirstTs, + lastMessageDate: orphanLastTs, + requests: orphanRequests, + launcherKind, + entrypoint: orphanEntrypoint, + }); + sessions.push(orphan); + } + } + + // Final pass: re-sort merged parent sessions' requests by timestamp and + // refresh the cached requestCount. + for (const session of sessions) { + if (session.requests.length > 1) { + session.requests.sort((a, b) => (a.timestamp ?? 0) - (b.timestamp ?? 0)); + } + session.requestCount = session.requests.length; + } + + return sessions.length > 0 ? { sessions, workspaceId, workspaceName } : null; +} + +export function parseClaudeSessions(projectsDir: string): { sessions: Session[]; workspaceId: string; workspaceName: string }[] { + const results: { sessions: Session[]; workspaceId: string; workspaceName: string }[] = []; + + let projectDirs: fs.Dirent[]; + try { + projectDirs = fs.readdirSync(projectsDir, { withFileTypes: true }).filter(e => e.isDirectory()); + } catch { + return results; + } + + for (const projDir of projectDirs) { + const result = parseClaudeProjectSessions(projectsDir, projDir.name); + if (result) results.push(result); + } + + return results; +} + +export async function parseClaudeSessionsAsync( + projectsDir: string, + onProject?: (idx: number, total: number, name: string) => void, +): Promise<{ sessions: Session[]; workspaceId: string; workspaceName: string }[]> { + const results: { sessions: Session[]; workspaceId: string; workspaceName: string }[] = []; + + let projectDirs: string[]; + try { + projectDirs = (await fs.promises.readdir(projectsDir, { withFileTypes: true })) + .filter(e => e.isDirectory()) + .map(e => e.name); + } catch { + return results; + } + + for (let i = 0; i < projectDirs.length; i++) { + const dirName = projectDirs[i]; + const workspaceName = projectNameFromEncoded(dirName, projectsDir); + + if (onProject) onProject(i + 1, projectDirs.length, workspaceName); + + const result = parseClaudeProjectSessions(projectsDir, dirName); + if (result) results.push(result); + + // Yield every 5 projects so the event loop stays responsive + if (i % 5 === 0) await new Promise(r => setTimeout(r, 0)); + } + + return results; +} + +function updateClaudeTimestampRange( + ts: number | null, + firstTs: number | null, + lastTs: number | null, +): { firstTs: number | null; lastTs: number | null } { + if (!ts) return { firstTs, lastTs }; + return { + firstTs: !firstTs || ts < firstTs ? ts : firstTs, + lastTs: !lastTs || ts > lastTs ? ts : lastTs, + }; +} + +function extractSkillsFromRefs(referencedFiles: string[]): string[] { + const skills = new Set(); + for (const ref of referencedFiles) { + const name = extractSkillNameFromPath(ref); + if (name) skills.add(name); + } + return [...skills]; +} + +function buildClaudeRequest( + line: ClaudeLine, + assistantData: ClaudeAssistantData, + userTs: number | null, + requestIndex: number, +): SessionRequest { + const hasAnyTokens = assistantData.assistantCount > 0; + const imageCount = countClaudeImages(line); + const uniqueRefs = [...new Set(assistantData.referencedFiles)]; + const skills = new Set(assistantData.skillsUsed); + for (const name of extractSkillsFromRefs(uniqueRefs)) skills.add(name); + return createRequest({ + requestId: line.uuid || `claude-${requestIndex}`, + timestamp: userTs, + messageText: getClaudeUserText(line), + responseText: assistantData.assistantTexts.join('\n'), + agentName: 'Claude', + agentMode: 'agent', + modelId: assistantData.model, + toolsUsed: assistantData.toolsUsed, + editedFiles: [...new Set(assistantData.editedFiles)], + referencedFiles: uniqueRefs, + skillsUsed: [...skills], + variableKinds: imageCount > 0 ? { image: imageCount } : {}, + totalElapsed: userTs && assistantData.lastTs ? assistantData.lastTs - userTs : null, + promptTokens: hasAnyTokens ? assistantData.totalInputTokens : null, + completionTokens: hasAnyTokens ? assistantData.totalOutputTokens : null, + cacheReadTokens: assistantData.totalCacheReadTokens > 0 ? assistantData.totalCacheReadTokens : null, + cacheWriteTokens: assistantData.totalCacheWriteTokens > 0 ? assistantData.totalCacheWriteTokens : null, + reasoningEffort: extractReasoningEffortFromModelId(assistantData.model), + }); +} + +function parseClaudeSessionFile(filePath: string, wsId: string, wsName: string): Session | null { + assertTrustedPath(filePath); + let raw: string; + try { + const content = readFileSafe(filePath); + if (content === null) return null; + raw = content; + } catch { + return null; + } + + const lines = parseClaudeLines(raw); + if (lines.length === 0) return null; + + const sessionId = lines[0].sessionId || path.basename(filePath, '.jsonl'); + const requests: SessionRequest[] = []; + let cwd = ''; + let firstTs: number | null = null; + let lastTs: number | null = null; + let entrypoint: string | undefined; + let i = 0; + + while (i < lines.length) { + const line = lines[i]; + if (line.type !== 'user' || !userHasText(line)) { + i++; + continue; + } + + if (!cwd && line.cwd) cwd = line.cwd; + // Capture the first non-empty entrypoint we see. In practice Claude Code + // writes it on every user line, but we only need one. + if (!entrypoint && line.entrypoint) entrypoint = line.entrypoint; + const userTs = getTimestamp(line.timestamp); + ({ firstTs, lastTs } = updateClaudeTimestampRange(userTs, firstTs, lastTs)); + + const assistantData = collectClaudeAssistantData(lines, i + 1, lastTs); + lastTs = assistantData.lastTs; + requests.push(buildClaudeRequest(line, assistantData, userTs, requests.length)); + i = assistantData.nextIndex; + } + + if (requests.length === 0) return null; + + const launcherKind = classifyLauncher(entrypoint); + return createSession({ + sessionId, + workspaceId: wsId, + workspaceName: wsName, + location: 'terminal', + harness: harnessForLauncher(launcherKind), + creationDate: firstTs, + lastMessageDate: lastTs, + requests, + hasDevcontainer: detectDevcontainerFromRequests(requests, cwd), + workspaceRootPath: cwd || undefined, + launcherKind, + entrypoint, + }); +} diff --git a/packages/ai-coach-analytics/src/parser-codex.ts b/packages/ai-coach-analytics/src/parser-codex.ts new file mode 100644 index 0000000..dc270d3 --- /dev/null +++ b/packages/ai-coach-analytics/src/parser-codex.ts @@ -0,0 +1,534 @@ +/*--------------------------------------------------------------------------------------------- + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the MIT License. See LICENSE in the project root for license information. + *--------------------------------------------------------------------------------------------*/ + +/* Codex CLI session parser + * + * Data layout (macOS): + * ~/.codex/sessions////rollout--.jsonl + * + * Each .jsonl file is a session. Lines have { type, timestamp, payload } + * + * type=session_meta: payload.id, payload.cwd, payload.cli_version, payload.source, payload.model_provider + * type=turn_context: payload.model, payload.effort + * type=event_msg: payload.type = user_message|agent_reasoning|function_call|function_call_output| + * task_started|task_complete|token_count|turn_aborted|error|assistant_message + * type=response_item: payload.role, payload.content[] + */ + +import * as fs from 'fs'; +import * as path from 'path'; +import { StringDecoder } from 'string_decoder'; +import { ModelUsage, Session, SessionRequest } from './types'; +import { assertTrustedPath, createRequest, createSession, detectDevcontainerFromRequests, extractSkillNameFromPath, extractSkillPathsFromText } from './parser-shared'; +import { canonicalizeReasoningEffort, extractReasoningEffortFromModelId } from './helpers'; + +interface CodexLine { + type: string; + timestamp?: string; + payload?: Record; +} + +interface CodexSessionMeta { + sessionId: string; + cwd: string; + source: string; + model: string; +} + +interface CodexContentItem { + type: string; + text?: string; +} + +interface CodexParseState { + requests: SessionRequest[]; + firstTs: number | null; + lastTs: number | null; + currentUserMessage: string; + currentAssistantTexts: string[]; + currentToolsUsed: string[]; + currentEditedFiles: string[]; + currentReferencedFiles: string[]; + currentSkillsUsed: string[]; + turnModel: string; + turnEffort: 'max' | 'high' | 'medium' | 'low' | null; + turnStartTs: number | null; + turnCanceled: boolean; + prevTotalInput: number; + prevTotalOutput: number; + curTotalInput: number; + curTotalOutput: number; + curTotalCachedInput: number; + hasTokenData: boolean; +} + +/** Tool names (lowercase) that actually write/edit files. */ +const CODEX_WRITE_TOOLS = new Set([ + 'write', 'write_file', 'create_file', 'edit', 'edit_file', + 'apply_diff', 'patch', 'multi_edit', 'create', 'overwrite', +]); + +function isRecord(value: unknown): value is Record { + return typeof value === 'object' && value !== null && !Array.isArray(value); +} + +function stringValue(value: unknown): string { + if (typeof value === 'string') return value; + if (typeof value === 'number' || typeof value === 'boolean') return String(value); + return ''; +} + +function recordValue(value: unknown): Record | undefined { + return isRecord(value) ? value : undefined; +} + +function parseCodexLine(rawLine: string): CodexLine | null { + try { + const parsed: unknown = JSON.parse(rawLine); + if (!isRecord(parsed) || typeof parsed.type !== 'string') return null; + let timestamp: string | undefined; + if (typeof parsed.timestamp === 'string') { + timestamp = parsed.timestamp; + } else if (typeof parsed.timestamp === 'number') { + timestamp = new Date(parsed.timestamp).toISOString(); + } + return { + type: parsed.type, + timestamp, + payload: recordValue(parsed.payload), + }; + } catch { + return null; + } +} + +function parseJsonRecord(raw: string): Record | null { + try { + const parsed: unknown = JSON.parse(raw); + return isRecord(parsed) ? parsed : null; + } catch { + return null; + } +} + +function projectNameFromCwd(cwd: string): string { + return cwd.replaceAll('\\', '/').replace(/\/+$/, '').split('/').pop() || 'unknown'; +} + +function createCodexState(initialModel: string): CodexParseState { + return { + requests: [], + firstTs: null, + lastTs: null, + currentUserMessage: '', + currentAssistantTexts: [], + currentToolsUsed: [], + currentEditedFiles: [], + currentReferencedFiles: [], + currentSkillsUsed: [], + turnModel: initialModel, + turnEffort: null, + turnStartTs: null, + turnCanceled: false, + prevTotalInput: 0, + prevTotalOutput: 0, + curTotalInput: 0, + curTotalOutput: 0, + curTotalCachedInput: 0, + hasTokenData: false, + }; +} + +function computeModelUsage(state: CodexParseState, model: string): Record | undefined { + if (!state.hasTokenData || (state.curTotalInput <= 0 && state.curTotalOutput <= 0)) return undefined; + const billingModel = state.turnModel || model || 'untracked'; + const uncachedInput = Math.max(0, state.curTotalInput - state.curTotalCachedInput); + return { + [billingModel]: { + inputTokens: uncachedInput, + outputTokens: state.curTotalOutput, + cacheReadTokens: state.curTotalCachedInput, + cacheWriteTokens: 0, + }, + }; +} + +function computeEndReason( + modelUsage: Record | undefined, + requests: SessionRequest[], +): 'shutdown' | 'aborted' | 'unknown' { + if (modelUsage) return 'unknown'; + const everyRequestEmpty = requests.every(request => + !request.responseText && request.toolsUsed.length === 0 && request.editedFiles.length === 0, + ); + return everyRequestEmpty ? 'aborted' : 'unknown'; +} + +function updateTimestamps(state: CodexParseState, ts: number | null): void { + if (!ts) return; + if (!state.firstTs || ts < state.firstTs) state.firstTs = ts; + if (!state.lastTs || ts > state.lastTs) state.lastTs = ts; +} + +function isTurnEmpty(state: CodexParseState): boolean { + return state.currentAssistantTexts.length === 0 + && state.currentToolsUsed.length === 0 + && state.currentEditedFiles.length === 0 + && !state.turnCanceled + && state.curTotalInput === state.prevTotalInput + && state.curTotalOutput === state.prevTotalOutput; +} + +function flushCodexTurn(state: CodexParseState, defaultModel: string): void { + if (!state.currentUserMessage && state.currentAssistantTexts.length === 0) return; + + const responseText = state.currentAssistantTexts.join('\n'); + const reqPromptTokens = state.hasTokenData && state.curTotalInput > state.prevTotalInput + ? state.curTotalInput - state.prevTotalInput + : null; + const reqCompletionTokens = state.hasTokenData && state.curTotalOutput > state.prevTotalOutput + ? state.curTotalOutput - state.prevTotalOutput + : null; + state.prevTotalInput = state.curTotalInput; + state.prevTotalOutput = state.curTotalOutput; + + state.requests.push(createRequest({ + requestId: `codex-${state.requests.length}`, + timestamp: state.turnStartTs, + messageText: state.currentUserMessage, + responseText, + isCanceled: state.turnCanceled, + agentName: 'Codex', + agentMode: 'agent', + modelId: state.turnModel || defaultModel, + toolsUsed: state.currentToolsUsed, + editedFiles: [...new Set(state.currentEditedFiles)], + referencedFiles: [...new Set(state.currentReferencedFiles)], + skillsUsed: [...new Set(state.currentSkillsUsed)], + totalElapsed: state.turnStartTs && state.lastTs ? state.lastTs - state.turnStartTs : null, + promptTokens: reqPromptTokens, + completionTokens: reqCompletionTokens, + reasoningEffort: state.turnEffort ?? extractReasoningEffortFromModelId(state.turnModel || defaultModel), + })); + + state.currentUserMessage = ''; + state.currentAssistantTexts = []; + state.currentToolsUsed = []; + state.currentEditedFiles = []; + state.currentReferencedFiles = []; + state.currentSkillsUsed = []; + state.turnStartTs = null; + state.turnCanceled = false; +} + +function extractContentItems(value: unknown): CodexContentItem[] { + if (!Array.isArray(value)) return []; + const items: CodexContentItem[] = []; + for (const item of value) { + if (!isRecord(item) || typeof item.type !== 'string') continue; + items.push({ + type: item.type, + text: typeof item.text === 'string' ? item.text : undefined, + }); + } + return items; +} + +/** Pull SKILL.md path references out of Codex function-call arguments, + * which are typically shell commands like `cat .../skills//SKILL.md`. + * Updates state.currentReferencedFiles and state.currentSkillsUsed. */ +function collectSkillsFromArgs(args: Record | null | undefined, state: CodexParseState): void { + if (!args) return; + const candidates: string[] = []; + for (const key of ['cmd', 'command', 'script', 'input']) { + const v = args[key]; + if (typeof v === 'string') candidates.push(v); + else if (Array.isArray(v)) { + for (const part of v) if (typeof part === 'string') candidates.push(part); + } + } + for (const text of candidates) { + for (const skillPath of extractSkillPathsFromText(text)) { + state.currentReferencedFiles.push(skillPath); + const name = extractSkillNameFromPath(skillPath); + if (name) state.currentSkillsUsed.push(name); + } + } +} + +function extractFilePath(args: Record | null | undefined): string | null { + if (!args) return null; + if (typeof args.file_path === 'string') return args.file_path; + if (typeof args.path === 'string') return args.path; + if (typeof args.filename === 'string') return args.filename; + return null; +} + +function handleUserMessageEvent(payload: Record, state: CodexParseState, ts: number | null, defaultModel: string): void { + const newMessage = stringValue(payload.message) || stringValue(payload.text); + if (state.currentUserMessage && state.currentUserMessage === newMessage && isTurnEmpty(state)) { + if (state.turnStartTs == null) state.turnStartTs = ts; + return; + } + + flushCodexTurn(state, defaultModel); + state.currentUserMessage = newMessage; + state.turnStartTs = ts; +} + +function handleFunctionCallEvent(payload: Record, state: CodexParseState): void { + const toolName = stringValue(payload.name) || 'unknown'; + state.currentToolsUsed.push(toolName); + + const args = recordValue(payload.arguments); + collectSkillsFromArgs(args, state); + + if (!CODEX_WRITE_TOOLS.has(toolName.toLowerCase())) return; + const filePath = extractFilePath(args); + if (filePath) state.currentEditedFiles.push(filePath); +} + +function handleAssistantMessageEvent(payload: Record, state: CodexParseState): void { + const content = payload.content; + if (typeof content === 'string') state.currentAssistantTexts.push(content); +} + +function handleTokenCountEvent(payload: Record, state: CodexParseState): void { + const info = recordValue(payload.info); + const totalUsage = recordValue(info?.total_token_usage); + if (!totalUsage) return; + + state.curTotalInput = typeof totalUsage.input_tokens === 'number' ? totalUsage.input_tokens : 0; + state.curTotalOutput = typeof totalUsage.output_tokens === 'number' ? totalUsage.output_tokens : 0; + state.curTotalCachedInput = typeof totalUsage.cached_input_tokens === 'number' ? totalUsage.cached_input_tokens : 0; + state.hasTokenData = true; +} + +function handleEventMsg(payload: Record, state: CodexParseState, ts: number | null, defaultModel: string): void { + const eventType = stringValue(payload.type); + if (eventType === 'user_message') { + handleUserMessageEvent(payload, state, ts, defaultModel); + return; + } + if (eventType === 'agent_reasoning') { + state.currentAssistantTexts.push(stringValue(payload.text)); + return; + } + if (eventType === 'function_call') { + handleFunctionCallEvent(payload, state); + return; + } + if (eventType === 'assistant_message') { + handleAssistantMessageEvent(payload, state); + return; + } + if (eventType === 'token_count') { + handleTokenCountEvent(payload, state); + return; + } + if (eventType === 'turn_aborted') state.turnCanceled = true; +} + +function handleTurnContext(payload: Record, state: CodexParseState): void { + const model = stringValue(payload.model); + if (model) state.turnModel = model; + const effort = payload.effort; + if (effort !== undefined && effort !== null) { + state.turnEffort = canonicalizeReasoningEffort(stringValue(effort)); + } +} + +function handleUserResponseItem(payload: Record, state: CodexParseState, ts: number | null, defaultModel: string): void { + for (const item of extractContentItems(payload.content)) { + if (item.type !== 'input_text' || !item.text || item.text.startsWith('<')) continue; + if (!state.currentUserMessage) { + flushCodexTurn(state, defaultModel); + state.currentUserMessage = item.text; + state.turnStartTs = ts; + } + } +} + +function handleAssistantResponseItem(payload: Record, state: CodexParseState): void { + for (const item of extractContentItems(payload.content)) { + if (item.type === 'output_text' && item.text) state.currentAssistantTexts.push(item.text); + } +} + +function handleFunctionCallResponseItem(payload: Record, state: CodexParseState): void { + const toolName = stringValue(payload.name); + if (!toolName) return; + + state.currentToolsUsed.push(toolName); + + const args = typeof payload.arguments === 'string' + ? parseJsonRecord(payload.arguments) + : recordValue(payload.arguments); + collectSkillsFromArgs(args, state); + + if (!CODEX_WRITE_TOOLS.has(toolName.toLowerCase())) return; + const filePath = extractFilePath(args); + if (filePath) state.currentEditedFiles.push(filePath); +} + +function handleResponseItem(payload: Record, state: CodexParseState, ts: number | null, defaultModel: string): void { + const role = stringValue(payload.role); + const itemType = stringValue(payload.type); + if (role === 'user') { + handleUserResponseItem(payload, state, ts, defaultModel); + return; + } + if (role === 'assistant' && itemType === 'message') { + handleAssistantResponseItem(payload, state); + return; + } + if (itemType === 'function_call') handleFunctionCallResponseItem(payload, state); +} + +function updateSessionMeta(line: CodexLine, meta: CodexSessionMeta): void { + if (line.type === 'session_meta') { + const payload = line.payload || {}; + meta.sessionId = stringValue(payload.id) || meta.sessionId; + meta.cwd = stringValue(payload.cwd) || meta.cwd; + meta.source = stringValue(payload.source) || meta.source; + } + if (line.type === 'turn_context' && !meta.model) { + meta.model = stringValue(line.payload?.model); + } +} + +function handleCodexLine(line: CodexLine, state: CodexParseState, meta: CodexSessionMeta): void { + updateSessionMeta(line, meta); + const ts = line.timestamp ? new Date(line.timestamp).getTime() : null; + updateTimestamps(state, ts); + + if (line.type === 'event_msg') { + handleEventMsg(line.payload || {}, state, ts, meta.model); + return; + } + if (line.type === 'turn_context') { + handleTurnContext(line.payload || {}, state); + return; + } + if (line.type === 'response_item') handleResponseItem(line.payload || {}, state, ts, meta.model); +} + +function readCodexJsonlStreaming(filePath: string, onLine: (line: CodexLine) => void): void { + const fd = fs.openSync(filePath, 'r'); + const decoder = new StringDecoder('utf8'); + const buffer = Buffer.allocUnsafe(1024 * 1024); + let remainder = ''; + + try { + while (true) { + const bytesRead = fs.readSync(fd, buffer, 0, buffer.length, null); + if (bytesRead === 0) break; + const text = remainder + decoder.write(buffer.subarray(0, bytesRead)); + let start = 0; + let nextNewline = text.indexOf('\n', start); + while (nextNewline !== -1) { + const rawLine = text.slice(start, nextNewline); + if (rawLine.trim()) { + const parsed = parseCodexLine(rawLine); + if (parsed) onLine(parsed); + } + start = nextNewline + 1; + nextNewline = text.indexOf('\n', start); + } + remainder = text.slice(start); + } + + remainder += decoder.end(); + if (remainder.trim()) { + const parsed = parseCodexLine(remainder); + if (parsed) onLine(parsed); + } + } finally { + fs.closeSync(fd); + } +} + +export function findCodexDirs(): string[] { + const home = process.env.HOME || process.env.USERPROFILE || ''; + const dirs: string[] = []; + for (const name of ['sessions', 'archived_sessions', 'archived-sessions']) { + const sessionsDir = path.join(home, '.codex', name); + if (fs.existsSync(sessionsDir)) dirs.push(sessionsDir); + } + return dirs; +} + +export function parseCodexSessions(sessionsDir: string): Session[] { + const sessions: Session[] = []; + const files = findAllJsonlFiles(sessionsDir); + + for (const filePath of files) { + const session = parseCodexSessionFile(filePath); + if (session) sessions.push(session); + } + + return sessions; +} + +function findAllJsonlFiles(dir: string): string[] { + const result: string[] = []; + try { + const entries = fs.readdirSync(dir, { withFileTypes: true }); + for (const e of entries) { + const full = path.join(dir, e.name); + if (e.isDirectory()) { + result.push(...findAllJsonlFiles(full)); + } else if (e.isFile() && e.name.endsWith('.jsonl')) { + result.push(full); + } + } + } catch { + /* skip unreadable dirs */ + } + return result; +} + +function parseCodexSessionFile(filePath: string): Session | null { + assertTrustedPath(filePath); + const meta: CodexSessionMeta = { sessionId: '', cwd: '', source: '', model: '' }; + const state = createCodexState(''); + let parsedLineCount = 0; + + try { + readCodexJsonlStreaming(filePath, (line) => { + parsedLineCount++; + handleCodexLine(line, state, meta); + }); + } catch { + return null; + } + + if (parsedLineCount === 0) return null; + if (!meta.sessionId) meta.sessionId = path.basename(filePath, '.jsonl'); + + const wsName = projectNameFromCwd(meta.cwd); + const wsId = `codex-${wsName}-${meta.sessionId.slice(0, 8)}`; + + flushCodexTurn(state, meta.model); + if (state.requests.length === 0) return null; + + const modelUsage = computeModelUsage(state, meta.model); + const endReason = computeEndReason(modelUsage, state.requests); + + return createSession({ + sessionId: meta.sessionId, + workspaceId: wsId, + workspaceName: wsName, + location: meta.source || 'terminal', + harness: 'Codex', + creationDate: state.firstTs, + lastMessageDate: state.lastTs, + requests: state.requests, + modelUsage, + endReason, + hasDevcontainer: detectDevcontainerFromRequests(state.requests, meta.cwd), + workspaceRootPath: meta.cwd || undefined, + }); +} diff --git a/packages/ai-coach-analytics/src/parser-shared.ts b/packages/ai-coach-analytics/src/parser-shared.ts new file mode 100644 index 0000000..658504b --- /dev/null +++ b/packages/ai-coach-analytics/src/parser-shared.ts @@ -0,0 +1,337 @@ +/*--------------------------------------------------------------------------------------------- + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the MIT License. See LICENSE in the project root for license information. + *--------------------------------------------------------------------------------------------*/ + +/* Shared parsing utilities used by all harness-specific parsers */ + +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeBlock, Session, SessionRequest, Workspace } from './types'; +import { SessionSource } from './cache'; +import { classifyWorkType } from './helpers'; +import { warnCore } from './log'; +import { SessionSchema } from './schemas'; + +/* ---- Path safety ---- */ + +/** + * Validates that a file path is within trusted directories and does not + * contain path traversal sequences. Throws if the path is unsafe. + */ +export function assertTrustedPath(filePath: string): void { + const normalized = path.resolve(filePath); + + // Reject path traversal + const segments = filePath.replaceAll('\\', '/').split('/'); + if (segments.includes('..')) { + throw new Error(`Path traversal detected: ${filePath}`); + } + + const trustedRoots = getTrustedRoots(); + const isTrusted = trustedRoots.some(root => normalized.startsWith(root + path.sep) || normalized === root); + if (!isTrusted) { + throw new Error(`Path is outside trusted directories: ${filePath}`); + } +} + +function getTrustedRoots(): string[] { + const roots: string[] = []; + const home = process.env.HOME || process.env.USERPROFILE || ''; + if (home) roots.push(path.resolve(home)); + + // VS Code extension storage paths + if (process.platform === 'darwin') { + if (home) roots.push(path.resolve(home, 'Library', 'Application Support', 'Code')); + if (home) roots.push(path.resolve(home, 'Library', 'Application Support', 'Code - Insiders')); + } else if (process.platform === 'win32') { + const appdata = process.env.APPDATA || ''; + if (appdata) { + roots.push(path.resolve(appdata, 'Code')); + roots.push(path.resolve(appdata, 'Code - Insiders')); + } + } else { + if (home) roots.push(path.resolve(home, '.config', 'Code')); + if (home) roots.push(path.resolve(home, '.config', 'Code - Insiders')); + } + + // Standard session log locations + if (home) { + roots.push(path.resolve(home, '.copilot')); + roots.push(path.resolve(home, '.claude')); + roots.push(path.resolve(home, '.codex')); + roots.push(path.resolve(home, '.local', 'share', 'opencode')); + roots.push(path.resolve(home, '.config', 'github-copilot')); + } + + // OS temp directory (used by tests and VS Code temp storage) + const tmpDir = os.tmpdir(); + if (tmpDir) roots.push(path.resolve(tmpDir)); + + return roots.filter(r => r.length > 0); +} + +/* ---- OOM protection ---- */ + +/** Maximum file size (50 MB) that parsers will read into memory. */ +export const MAX_FILE_SIZE = 50 * 1024 * 1024; + +/** + * Reads a file safely, returning null (with a warning) if the file exceeds MAX_FILE_SIZE. + */ +export function readFileSafe(filePath: string): string | null { + try { + const stat = fs.statSync(filePath); + if (stat.size > MAX_FILE_SIZE) { + warnCore('parser', `Skipping oversized file (${(stat.size / 1024 / 1024).toFixed(1)} MB): ${filePath}`); + return null; + } + } catch (e) { + warnCore('parser', `Cannot stat file: ${filePath}`, e); + return null; + } + return fs.readFileSync(filePath, 'utf-8'); +} + +/** Shared prefetch cache: file path -> contents. Populated async, consumed sync by parsers. */ +export const prefetchCache = new Map(); + +export const CODE_BLOCK_RE = /```(\w+)?\n([\s\S]*?)```/g; + +const MAX_STORED_MESSAGE_CHARS = 16_000; +const MAX_STORED_RESPONSE_CHARS = 24_000; +const MAX_CODE_SCAN_CHARS = 128_000; + +export const LANG_ALIASES: Record = { + sh: 'bash', shell: 'bash', zsh: 'bash', + ts: 'typescript', tsx: 'typescript', + js: 'javascript', jsx: 'javascript', + py: 'python', python3: 'python', + cs: 'csharp', 'c#': 'csharp', + yml: 'yaml', md: 'markdown', + tf: 'terraform', rs: 'rust', rb: 'ruby', + jsonc: 'json', jsonl: 'json', + txt: 'text', plaintext: 'text', env: 'dotenv', +}; + +const EXT_TO_TECH: Record = { + py: 'Python', pyx: 'Python', pyi: 'Python', + js: 'JavaScript', mjs: 'JavaScript', cjs: 'JavaScript', + ts: 'TypeScript', tsx: 'TypeScript', jsx: 'React', + java: 'Java', cs: 'C#', csx: 'C#', + go: 'Go', rs: 'Rust', rb: 'Ruby', php: 'PHP', + swift: 'Swift', kt: 'Kotlin', kts: 'Kotlin', scala: 'Scala', + c: 'C', h: 'C', cpp: 'C++', cc: 'C++', cxx: 'C++', hpp: 'C++', + html: 'HTML', htm: 'HTML', + css: 'CSS', scss: 'CSS', sass: 'CSS', less: 'CSS', + json: 'JSON', jsonc: 'JSON', jsonl: 'JSON', + yaml: 'YAML', yml: 'YAML', toml: 'TOML', xml: 'XML', + sql: 'SQL', sh: 'Shell', bash: 'Shell', zsh: 'Shell', + ps1: 'PowerShell', psm1: 'PowerShell', + tf: 'Terraform', tfvars: 'Terraform', bicep: 'Bicep', + md: 'Markdown', mdx: 'Markdown', + dockerfile: 'Docker', r: 'R', lua: 'Lua', dart: 'Dart', + vue: 'Vue', svelte: 'Svelte', ipynb: 'Jupyter', + proto: 'Protobuf', graphql: 'GraphQL', gql: 'GraphQL', + python: 'Python', javascript: 'JavaScript', typescript: 'TypeScript', + csharp: 'C#', rust: 'Rust', ruby: 'Ruby', + terraform: 'Terraform', text: 'Text', dotenv: 'Dotenv', + mermaid: 'Mermaid', tex: 'LaTeX', latex: 'LaTeX', + properties: 'Properties', ini: 'INI', + gitignore: 'Git Config', ignore: 'Git Config', dockerignore: 'Docker', + powershell: 'PowerShell', hcl: 'Terraform', + log: 'Log', diff: 'Diff', csv: 'CSV', svg: 'XML', + console: 'Shell', azurecli: 'Shell', dotnetcli: 'Shell', + makefile: 'Make', bicepparam: 'Bicep', +}; + +export function techFromPath(filePath: string): string { + const name = filePath.replaceAll('\\', '/').split('/').pop()!.toLowerCase(); + if (name === 'dockerfile' || name.startsWith('dockerfile.')) return 'Docker'; + if (name === 'makefile' || name === 'gnumakefile') return 'Make'; + if (name === 'cmakelists.txt') return 'CMake'; + const ext = name.includes('.') ? name.split('.').pop()! : ''; + return EXT_TO_TECH[ext] || ''; +} + +export function shortPath(fullPath: string, workspaceName: string): string { + const parts = fullPath.replaceAll('\\', '/').split('/'); + for (let i = 0; i < parts.length; i++) { + if (parts[i] === workspaceName) { + return parts.slice(i + 1).join('/'); + } + } + return parts.length > 3 ? parts.slice(-3).join('/') : fullPath; +} + +export function extractCodeBlocks(text: string): CodeBlock[] { + const blocks: CodeBlock[] = []; + let m: RegExpExecArray | null; + const re = new RegExp(CODE_BLOCK_RE.source, 'g'); + while ((m = re.exec(text)) !== null) { + let lang = (m[1] || 'unknown').toLowerCase().trim(); + lang = LANG_ALIASES[lang] || lang; + const code = m[2].trim(); + const loc = code ? code.split('\n').length : 0; + blocks.push({ language: lang, loc }); + } + return blocks; +} + +function compactTextForStorage(text: string, maxChars: number): string { + if (text.length <= maxChars) return text; + const headChars = Math.max(0, Math.floor(maxChars * 0.75)); + const tailChars = Math.max(0, maxChars - headChars - 64); + const omitted = text.length - headChars - tailChars; + const marker = `\n\n[truncated ${omitted.toLocaleString()} chars]\n\n`; + return text.slice(0, headChars) + marker + text.slice(text.length - tailChars); +} + +function textForCodeScan(text: string): string { + if (text.length <= MAX_CODE_SCAN_CHARS) return text; + return text.slice(0, MAX_CODE_SCAN_CHARS); +} + +/* ---- Factories ---- */ + +/** Context object passed through all directory-level parsing operations. */ +export interface ParseContext { + workspaces: Map; + sessions: Session[]; + editLocIndex: Map>; + sessionSourceIndex: Map; + /** Running total of AI-generated lines of code (sum of all aiCode blocks). */ + aiLoc: number; +} + +/** Creates a SessionRequest with sensible defaults; callers only supply non-default fields. */ +export function createRequest(overrides: Partial & Pick): SessionRequest { + const rawMsg = overrides.messageText; + const rawResp = overrides.responseText; + const msg = compactTextForStorage(rawMsg, MAX_STORED_MESSAGE_CHARS); + const resp = compactTextForStorage(rawResp, MAX_STORED_RESPONSE_CHARS); + const sanitizedTs = (overrides.timestamp != null && overrides.timestamp > 0) ? overrides.timestamp : null; + const { timestamp: _ts, ...rest } = overrides; + return { + requestId: '', + timestamp: sanitizedTs, + isCanceled: false, + agentName: '', + agentMode: '', + modelId: '', + toolsUsed: [], + editedFiles: [], + referencedFiles: [], + slashCommand: '', + variableKinds: {}, + customInstructions: [], + skillsUsed: [], + firstProgress: null, + totalElapsed: null, + toolConfirmations: [], + promptTokens: null, + completionTokens: null, + cacheReadTokens: null, + cacheWriteTokens: null, + compaction: null, + todoSnapshot: null, + reasoningEffort: null, + ...rest, + messageText: msg, + responseText: resp, + messageLength: rawMsg.length, + responseLength: rawResp.length, + userCode: overrides.userCode ?? extractCodeBlocks(textForCodeScan(rawMsg)), + aiCode: overrides.aiCode ?? extractCodeBlocks(textForCodeScan(rawResp)), + workType: overrides.workType || classifyWorkType(msg), + }; +} + +/** Creates a Session with sensible defaults; callers only supply non-default fields. */ +export function createSession(overrides: Partial & Pick): Session { + const reqs = overrides.requests; + const timestamps = reqs.map(r => r.timestamp).filter((t): t is number => t != null && t > 0); + const computed = { + creationDate: timestamps.length > 0 ? Math.min(...timestamps) : null, + lastMessageDate: timestamps.length > 0 ? Math.max(...timestamps) : null, + }; + const merged = { + location: 'panel' as const, + ...computed, + requestCount: reqs.length, + ...overrides, + }; + // Sanitize session-level timestamps that came from overrides (e.g. raw file data with 0). + if (merged.creationDate != null && merged.creationDate <= 0) merged.creationDate = computed.creationDate; + if (merged.lastMessageDate != null && merged.lastMessageDate <= 0) merged.lastMessageDate = computed.lastMessageDate; + return merged; +} + +const DEVCONTAINER_PATH_RE = /(?:^|[\s=:"'`])\/workspaces\//; + +export function detectDevcontainerFromRequests(requests: SessionRequest[], cwd?: string): boolean { + if (typeof cwd === 'string' && cwd.startsWith('/workspaces/')) return true; + for (const r of requests) { + if (Array.isArray(r.toolConfirmations)) { + for (const tc of r.toolConfirmations) { + if (tc.isTerminal && typeof tc.commandLine === 'string' && DEVCONTAINER_PATH_RE.test(tc.commandLine)) { + return true; + } + } + } + if (Array.isArray(r.editedFiles)) { + for (const f of r.editedFiles) { + if (typeof f === 'string' && f.startsWith('/workspaces/')) return true; + } + } + if (Array.isArray(r.referencedFiles)) { + for (const f of r.referencedFiles) { + if (typeof f === 'string' && f.startsWith('/workspaces/')) return true; + } + } + } + return false; +} + +/** Regex to extract the skill directory name from a path ending in `/skills//SKILL.md`. */ +const SKILL_PATH_RE = /[/\\]skills[/\\]([^/\\]+)[/\\]SKILL\.md$/i; + +/** Extract a skill name from a file path pointing to a SKILL.md file. + * Returns null if the path does not match the expected pattern. */ +export function extractSkillNameFromPath(rawPath: string): string | null { + const m = SKILL_PATH_RE.exec(rawPath); + if (!m) return null; + const name = m[1].trim(); + return (name && !name.includes('ai_toolkit')) ? name : null; +} + +/** Regex to find SKILL.md path references embedded inside a larger string + * (e.g. a shell command line like `cat ~/.codex/skills/foo/SKILL.md`). + * The skill segment forbids whitespace and shell quote chars so the match + * cannot run past the path boundary. Slug-style names (the convention in + * the wild) are not affected by these restrictions. */ +const SKILL_PATH_IN_TEXT_RE = /[/\\]skills[/\\][^\s/\\'"`]+[/\\]SKILL\.md/gi; + +/** Extract all SKILL.md path references embedded in a string. Use this for + * harnesses whose tool calls expose shell commands or other free-form text + * rather than structured `file_path` arguments (e.g. Codex). For structured + * paths, prefer {@link extractSkillNameFromPath} directly. */ +export function extractSkillPathsFromText(text: string): string[] { + if (typeof text !== 'string' || !text) return []; + return text.match(SKILL_PATH_IN_TEXT_RE) || []; +} + +/** + * Optional validation gate that individual parsers can call before returning + * sessions. Returns the session unchanged if valid, or null (with a warning) + * if the session is malformed. Non-breaking: parsers may choose to skip this. + */ +export function validateSession(session: unknown, source: string): Session | null { + const result = SessionSchema.safeParse(session); + if (result.success) { + return result.data as Session; + } + warnCore('schema', `Invalid session from ${source}: ${result.error.issues[0]?.message ?? 'unknown'}`); + return null; +} diff --git a/packages/ai-coach-analytics/src/profanity.ts b/packages/ai-coach-analytics/src/profanity.ts new file mode 100644 index 0000000..113245f --- /dev/null +++ b/packages/ai-coach-analytics/src/profanity.ts @@ -0,0 +1,46 @@ +/*--------------------------------------------------------------------------------------------- + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the MIT License. See LICENSE in the project root for license information. + *--------------------------------------------------------------------------------------------*/ + +/** + * Profanity detection backed by the `leo-profanity` dictionary, so the + * plaintext wordlist lives in an external package and is not committed to + * this repository. + * + * We strip fenced code blocks and inline backticks before checking so that + * strings inside code snippets don't produce false positives. + */ + +import leoProfanity from 'leo-profanity'; + +/** Strip code blocks and inline-backtick content so we don't flag code strings. */ +function stripCode(text: string): string { + return text.replaceAll(/```[\s\S]*?```/g, '').replaceAll(/`[^`]+`/g, ''); +} + +/** Returns true if the (code-stripped) text contains any flagged word. */ +export function containsProfanity(text: string): boolean { + if (!text) return false; + return leoProfanity.check(stripCode(text)); +} + +/** + * Extract all flagged words found in the text (deduped, order-preserving, + * lowercased). Matches inside fenced or inline code are ignored. + */ +export function extractProfaneWords(text: string): string[] { + if (!text) return []; + const cleaned = stripCode(text); + const found = leoProfanity.badWordsUsed(cleaned); + const seen = new Set(); + const out: string[] = []; + for (const raw of found) { + const w = raw.toLowerCase(); + if (!seen.has(w)) { + seen.add(w); + out.push(w); + } + } + return out; +} diff --git a/packages/ai-coach-analytics/src/rule-engine.ts b/packages/ai-coach-analytics/src/rule-engine.ts new file mode 100644 index 0000000..9eb6b64 --- /dev/null +++ b/packages/ai-coach-analytics/src/rule-engine.ts @@ -0,0 +1,322 @@ +/*--------------------------------------------------------------------------------------------- + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the MIT License. See LICENSE in the project root for license information. + *--------------------------------------------------------------------------------------------*/ + +/** + * Rule engine: loads detection rules from markdown files and bridges them with + * the existing detector functions. Each rule's metadata (thresholds, description + * templates, severity) is configurable, while the detection logic delegates to + * the battle-tested detector implementations. + * + * This module also supports evaluating rules against live data for the preview + * feature in the Rule Editor page. + */ + +import { DetectionRule, DetectorEmission, RulePreviewStats, RuleEvalResult, RuleTemplateVars, AntiPattern, PracticeGroup, Session, SessionRequest } from './types'; +import { parseRule, fillTemplate, serializeRule } from './rule-parser'; +import { warnCore } from './log'; + +/* ---- Multi-layer rule storage ---- */ + +// Rule markdown sources keyed by rule ID, per layer. +const BUILTIN_RULE_SOURCES: Record = {}; +const PERSONAL_RULE_SOURCES: Record = {}; +const PROJECT_RULE_SOURCES: Record = {}; + +// Parsed rule caches per layer (null = not yet parsed). +let builtinRules: DetectionRule[] | null = null; +let personalRules: DetectionRule[] | null = null; +let projectRules: DetectionRule[] | null = null; + +// In-memory overrides (e.g. threshold tweaks from the UI). +const inMemoryOverrides: DetectionRule[] = []; + +/** + * Register a built-in rule source. Called during module initialization. + */ +export function registerBuiltinRuleSource(id: string, source: string): void { + BUILTIN_RULE_SOURCES[id] = source; +} + +/** + * Register a personal rule source (from ~/.ai-engineer-coach/rules/). + */ +export function registerPersonalRuleSource(id: string, source: string, filePath: string): void { + PERSONAL_RULE_SOURCES[id] = { source, filePath }; + personalRules = null; // invalidate cache +} + +/** + * Register a project rule source (from /.ai-engineer-coach/rules/). + */ +export function registerProjectRuleSource(id: string, source: string, filePath: string): void { + PROJECT_RULE_SOURCES[id] = { source, filePath }; + projectRules = null; // invalidate cache +} + +/** + * Clear all personal/project rules (for reload). + */ +export function clearLayerRules(layer: 'personal' | 'project'): void { + if (layer === 'personal') { + for (const k of Object.keys(PERSONAL_RULE_SOURCES)) delete PERSONAL_RULE_SOURCES[k]; + personalRules = null; + } else { + for (const k of Object.keys(PROJECT_RULE_SOURCES)) delete PROJECT_RULE_SOURCES[k]; + projectRules = null; + } +} + +/** + * Load and parse all built-in rule sources. Cached after first call. + */ +export function loadBuiltinRules(): DetectionRule[] { + if (builtinRules) return builtinRules; + builtinRules = []; + for (const [id, source] of Object.entries(BUILTIN_RULE_SOURCES)) { + const rule = parseRule(source); + if (rule) { + rule.source = 'built-in'; + rule.sourceFilePath = ''; + builtinRules.push(rule); + } else { + warnCore('RuleEngine', `Failed to parse built-in rule: ${id}`); + } + } + return builtinRules; +} + +function loadPersonalRules(): DetectionRule[] { + if (personalRules) return personalRules; + personalRules = []; + for (const [id, { source, filePath }] of Object.entries(PERSONAL_RULE_SOURCES)) { + const rule = parseRule(source); + if (rule) { + rule.source = 'personal'; + rule.sourceFilePath = filePath; + personalRules.push(rule); + } else { + warnCore('RuleEngine', `Failed to parse personal rule: ${id}`); + } + } + return personalRules; +} + +function loadProjectRules(): DetectionRule[] { + if (projectRules) return projectRules; + projectRules = []; + for (const [id, { source, filePath }] of Object.entries(PROJECT_RULE_SOURCES)) { + const rule = parseRule(source); + if (rule) { + rule.source = 'project'; + rule.sourceFilePath = filePath; + projectRules.push(rule); + } else { + warnCore('RuleEngine', `Failed to parse project rule: ${id}`); + } + } + return projectRules; +} + +/** + * Get all rules with 3-tier precedence: built-in < personal < project. + * In-memory overrides (from threshold editing) take highest precedence. + */ +export function getAllRules(): DetectionRule[] { + const builtin = loadBuiltinRules(); + const personal = loadPersonalRules(); + const project = loadProjectRules(); + + // Merge: project > personal > built-in. In-memory overrides win over everything. + const merged = new Map(); + for (const r of builtin) merged.set(r.id, r); + for (const r of personal) merged.set(r.id, r); + for (const r of project) merged.set(r.id, r); + for (const r of inMemoryOverrides) merged.set(r.id, r); + + return [...merged.values()]; +} + +/** + * Get a single rule by ID. + */ +export function getRule(id: string): DetectionRule | undefined { + return getAllRules().find(r => r.id === id); +} + +/** + * Add or update an in-memory rule override. + */ +export function setUserRule(rule: DetectionRule): void { + if (rule.source === 'built-in') { + rule.source = 'personal'; + } + const idx = inMemoryOverrides.findIndex(r => r.id === rule.id); + if (idx >= 0) inMemoryOverrides[idx] = rule; + else inMemoryOverrides.push(rule); +} + +/** + * Remove an in-memory rule override. Layer rules (file-based) cannot be removed this way. + */ +export function removeUserRule(id: string): boolean { + const idx = inMemoryOverrides.findIndex(r => r.id === id); + if (idx < 0) return false; + inMemoryOverrides.splice(idx, 1); + return true; +} + +/** + * Update thresholds on a rule (user override of built-in). + */ +export function updateRuleThresholds(id: string, thresholds: Record): DetectionRule | null { + const existing = getRule(id); + if (!existing) return null; + + const updated: DetectionRule = { + ...existing, + thresholds: { ...existing.thresholds, ...thresholds }, + source: existing.source === 'built-in' ? 'personal' : existing.source, + rawSource: '', // Will be regenerated + }; + updated.rawSource = serializeRule(updated); + setUserRule(updated); + return updated; +} + +/** + * Create a new rule from markdown source. + */ +export function createRuleFromMarkdown(markdown: string): DetectionRule | null { + const rule = parseRule(markdown); + if (!rule) return null; + rule.source = 'personal'; + rule.sourceFilePath = ''; + setUserRule(rule); + return rule; +} + +/** + * Get rule markdown source for editing. + */ +export function getRuleSource(id: string): string | null { + const rule = getRule(id); + if (!rule) return null; + return rule.rawSource || serializeRule(rule); +} + +/** + * Evaluate a single rule against emission data. + */ +export function evaluateRule( + rule: DetectionRule, + emission: DetectorEmission | null, +): RuleEvalResult { + if (!emission || emission.count === 0) { + return { + ruleId: rule.id, + triggered: false, + occurrences: 0, + total: 0, + ratio: 0, + severity: rule.severity, + description: '', + suggestion: '', + examples: [], + templateVars: { count: 0, total: 0, ratio: 0, pct: '0%', extra: {} }, + }; + } + + const pct = emission.total > 0 + ? `${(emission.count / emission.total * 100).toFixed(0)}%` + : '0%'; + + const vars: RuleTemplateVars = { + count: emission.count, + total: emission.total, + ratio: emission.ratio, + pct, + extra: emission.extra, + }; + + const severity = emission.dynamicSeverity ?? rule.severity; + const description = rule.descriptionTemplate + ? fillTemplate(rule.descriptionTemplate, vars as unknown as Record) + : `${emission.count} occurrences`; + const suggestion = rule.suggestionTemplate + ? fillTemplate(rule.suggestionTemplate, vars as unknown as Record) + : ''; + + return { + ruleId: rule.id, + triggered: true, + occurrences: emission.count, + total: emission.total, + ratio: emission.ratio, + severity, + description, + suggestion, + examples: emission.examples, + templateVars: vars, + }; +} + +/** + * Get preview stats for all rules against current data. + * Used by the Rule Editor to show how many items each rule would flag. + */ +export function getRulePreviewStats( + reqs: SessionRequest[], + sessions: Session[], + skipIdeDetectors: boolean, + detectorResults: AntiPattern[], + emissions?: Map, +): RulePreviewStats[] { + const allRules = getAllRules(); + const resultMap = new Map(); + for (const r of detectorResults) resultMap.set(r.id, r); + + return allRules + .filter(rule => !skipIdeDetectors || !rule.requiresIdeContext) + .map(rule => { + const emission = emissions?.get(rule.id); + const result = resultMap.get(rule.id); + const total = emission?.total ?? (rule.scope === 'sessions' ? sessions.length : reqs.length); + const occurrences = emission?.count ?? result?.occurrences ?? 0; + const pct = total > 0 ? (occurrences / total) * 100 : 0; + + return { + ruleId: rule.id, + ruleName: rule.name, + triggered: !!result, + occurrences, + total, + pct: Math.round(pct * 10) / 10, + severity: rule.severity, + group: rule.group, + previewDescription: result?.description ?? 'Not triggered with current data.', + previewExamples: result?.examples ?? [], + }; + }); +} + +/** + * Get all rules grouped by practice group for UI display. + */ +export function getRulesGrouped(): Record { + const groups: Record = { + 'prompt-quality': [], + 'session-hygiene': [], + 'code-review': [], + 'tool-mastery': [], + 'context-management': [], + }; + for (const rule of getAllRules()) { + groups[rule.group].push(rule); + } + return groups; +} + +// Re-export for convenience +export { serializeRule, parseRule, fillTemplate }; diff --git a/packages/ai-coach-analytics/src/rule-loader.ts b/packages/ai-coach-analytics/src/rule-loader.ts new file mode 100644 index 0000000..985f732 --- /dev/null +++ b/packages/ai-coach-analytics/src/rule-loader.ts @@ -0,0 +1,503 @@ +/*--------------------------------------------------------------------------------------------- + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the MIT License. See LICENSE in the project root for license information. + *--------------------------------------------------------------------------------------------*/ + +/** + * Multi-layer rule loader. + * + * Rules are loaded from three directories (lowest to highest precedence): + * 1. Built-in: dist/rules/ (shipped with the extension) + * 2. Personal: ~/.ai-engineer-coach/rules/ (user-level, shared across workspaces) + * 3. Project: /.ai-engineer-coach/rules/ (workspace-specific) + * + * The rule engine merges them so that project rules override personal rules + * which override built-in rules (by matching on rule ID). + */ + +import * as fs from 'fs'; +import * as path from 'path'; +import * as os from 'os'; +import { fileURLToPath } from 'url'; +import { + registerBuiltinRuleSource, + registerPersonalRuleSource, + registerProjectRuleSource, + clearLayerRules, + loadBuiltinRules, +} from './rule-engine'; +import { parseMetric, registerMetric, clearMetrics } from './metric-engine'; +import type { MetricDefinition } from './metric-engine'; +import { warnCore } from './log'; +import { hashContent, type TrustGate, type TrustLayer } from './rule-trust'; + +const __dirname = path.dirname(fileURLToPath(import.meta.url)); + +let defaultTrustGate: TrustGate | undefined; +export function setDefaultTrustGate(gate: TrustGate | undefined): void { + defaultTrustGate = gate; +} +function resolveGate(explicit?: TrustGate): TrustGate | undefined { + return explicit ?? defaultTrustGate; +} + +/** + * Whether the current workspace is trusted (VS Code Workspace Trust). + * Project-layer rules and metrics execute DSL sourced from the repository, + * so they must never load in an untrusted workspace. The extension wires + * this to `vscode.workspace.isTrusted`; tests and scripts (this module has + * no vscode dependency) default to trusted. + */ +let workspaceTrustProvider: () => boolean = () => true; +export function setWorkspaceTrustProvider(provider: () => boolean): void { + workspaceTrustProvider = provider; +} +function isWorkspaceTrusted(): boolean { + return workspaceTrustProvider(); +} + +/** Hard cap on rule/metric files loaded from a single directory. */ +const MAX_FILES_PER_DIR = 200; + +function capFiles(files: string[], dir: string, label: string): string[] { + if (files.length <= MAX_FILES_PER_DIR) return files; + warnCore(label, `${dir} contains ${files.length} files; loading only the first ${MAX_FILES_PER_DIR}`); + return files.slice(0, MAX_FILES_PER_DIR); +} + +/** Well-known directory name used for personal and project rule storage. */ +export const RULES_DIR_NAME = '.ai-engineer-coach/rules'; + +/** Absolute path to the personal rules directory. */ +export function getPersonalRulesDir(): string { + return path.join(os.homedir(), RULES_DIR_NAME); +} + +/** Absolute path to the project rules directory for a given workspace root. */ +export function getProjectRulesDir(workspaceRoot: string): string { + return path.join(workspaceRoot, RULES_DIR_NAME); +} + +let builtinRegistered = false; + +/** + * Load all .md files from the built-in rules/ directory and register them. + * Safe to call multiple times (only loads once). + */ +export function registerAllBuiltinRules(): void { + if (builtinRegistered) return; + builtinRegistered = true; + + const rulesDir = path.join(__dirname, 'rules'); + let files: string[]; + try { + files = fs.readdirSync(rulesDir).filter(f => f.endsWith('.md')); + } catch { + const srcRulesDir = path.join(__dirname, '..', 'src', 'core', 'rules'); + try { + files = fs.readdirSync(srcRulesDir).filter(f => f.endsWith('.md')); + for (const file of files) { + const id = file.replace(/\.md$/, ''); + const source = fs.readFileSync(path.join(srcRulesDir, file), 'utf-8'); + registerBuiltinRuleSource(id, source); + } + return; + } catch { + warnCore('RuleLoader', 'Could not find rules directory'); + return; + } + } + + for (const file of files) { + const id = file.replace(/\.md$/, ''); + const source = fs.readFileSync(path.join(rulesDir, file), 'utf-8'); + registerBuiltinRuleSource(id, source); + } +} + +/** + * Load personal rules from ~/.ai-engineer-coach/rules/. + * Can be called multiple times to reload (clears + re-registers). + */ +export function loadPersonalRules(trustGate?: TrustGate): number { + clearLayerRules('personal'); + const dir = getPersonalRulesDir(); + return loadRulesFromDir(dir, (id, source, filePath) => registerPersonalRuleSource(id, source, filePath), + resolveGate(trustGate), 'personal'); +} + +/** + * Load project rules from /.ai-engineer-coach/rules/. + * Can be called multiple times to reload (clears + re-registers). + */ +export function loadProjectRules(workspaceRoot: string, trustGate?: TrustGate): number { + clearLayerRules('project'); + const dir = getProjectRulesDir(workspaceRoot); + return loadRulesFromDir(dir, (id, source, filePath) => registerProjectRuleSource(id, source, filePath), + resolveGate(trustGate), 'project'); +} + +/** + * Load all three layers. Call during extension activation. + */ +export function loadAllRuleLayers(workspaceRoot?: string, trustGate?: TrustGate): { builtin: number; personal: number; project: number } { + registerAllBuiltinRules(); + const personal = loadPersonalRules(trustGate); + const project = workspaceRoot ? loadProjectRules(workspaceRoot, trustGate) : 0; + return { + builtin: loadBuiltinRules().length, + personal, + project, + }; +} + +/** + * Scan a directory for .md rule files and call the register callback for each. + * Returns the number of rules loaded. + * + * When `trustGate` is supplied, each file is checked before registration. + * Files that fail the gate are skipped and reported via `trustGate.onBlocked`. + */ +function loadRulesFromDir( + dir: string, + register: (id: string, source: string, filePath: string) => void, + trustGate?: TrustGate, + layer: TrustLayer = 'personal', +): number { + let files: string[]; + try { + files = fs.readdirSync(dir).filter(f => f.endsWith('.md')); + } catch { + return 0; // Directory doesn't exist yet -- that's fine + } + if (layer === 'project' && !isWorkspaceTrusted()) { + warnCore('RuleLoader', 'Workspace is not trusted; skipping project rules'); + return 0; + } + files = capFiles(files, dir, 'RuleLoader'); + + let count = 0; + for (const file of files) { + const filePath = path.join(dir, file); + try { + const id = file.replace(/\.md$/, ''); + const source = fs.readFileSync(filePath, 'utf-8'); + if (trustGate && !trustGate.isAllowed(filePath, source)) { + trustGate.onBlocked({ filePath, layer, kind: 'rule', hash: hashContent(source), content: source }); + continue; + } + register(id, source, filePath); + count++; + } catch { + warnCore('RuleLoader', `Failed to read ${filePath}`); + } + } + return count; +} + +/** + * Summary of loaded rule layers (used by the UI help panel). + */ +export interface RuleLayerInfo { + layer: 'built-in' | 'personal' | 'project'; + directory: string; + exists: boolean; + ruleCount: number; +} + +/** + * Get info about all rule layers, useful for the UI help panel. + */ +export function getRuleLayerInfo(workspaceRoot?: string): RuleLayerInfo[] { + const builtinDir = path.join(__dirname, 'rules'); + const personalDir = getPersonalRulesDir(); + const projectDir = workspaceRoot ? getProjectRulesDir(workspaceRoot) : ''; + + function countMdFiles(dir: string): { exists: boolean; count: number } { + try { + const files = fs.readdirSync(dir).filter(f => f.endsWith('.md')); + return { exists: true, count: files.length }; + } catch { + return { exists: false, count: 0 }; + } + } + + const b = countMdFiles(builtinDir); + const p = countMdFiles(personalDir); + const pr = projectDir ? countMdFiles(projectDir) : { exists: false, count: 0 }; + + const layers: RuleLayerInfo[] = [ + { layer: 'built-in', directory: builtinDir, exists: b.exists, ruleCount: b.count }, + { layer: 'personal', directory: personalDir, exists: p.exists, ruleCount: p.count }, + ]; + if (workspaceRoot) { + layers.push({ layer: 'project', directory: projectDir, exists: pr.exists, ruleCount: pr.count }); + } + return layers; +} + +/* ================================================================== */ +/* Metric file loading (.metric.md) */ +/* ================================================================== */ + +let builtinMetricsRegistered = false; + +/** + * Load built-in .metric.md files from the metrics/ directory. + */ +export function registerAllBuiltinMetrics(): number { + if (builtinMetricsRegistered) return 0; + builtinMetricsRegistered = true; + + // Try dist/metrics/ first, then src/core/metrics/ + const dirs = [ + path.join(__dirname, 'metrics'), + path.join(__dirname, '..', 'src', 'core', 'metrics'), + ]; + + for (const dir of dirs) { + const count = loadMetricsFromDir(dir, 'built-in'); + if (count > 0) return count; + } + return 0; +} + +/** + * Load personal .metric.md files from ~/.ai-engineer-coach/metrics/. + */ +export function loadPersonalMetrics(trustGate?: TrustGate): number { + const dir = path.join(os.homedir(), '.ai-engineer-coach', 'metrics'); + return loadMetricsFromDir(dir, 'personal', resolveGate(trustGate), 'personal'); +} + +/** + * Load project .metric.md files from /.ai-engineer-coach/metrics/. + */ +export function loadProjectMetrics(workspaceRoot: string, trustGate?: TrustGate): number { + const dir = path.join(workspaceRoot, '.ai-engineer-coach', 'metrics'); + return loadMetricsFromDir(dir, 'project', resolveGate(trustGate), 'project'); +} + +/** + * Load all metric layers. Call during extension activation. + */ +export function loadAllMetricLayers(workspaceRoot?: string, trustGate?: TrustGate): { builtin: number; personal: number; project: number } { + clearMetrics(); + const builtin = registerAllBuiltinMetrics(); + const personal = loadPersonalMetrics(trustGate); + const project = workspaceRoot ? loadProjectMetrics(workspaceRoot, trustGate) : 0; + return { builtin, personal, project }; +} + +function loadMetricsFromDir( + dir: string, + source: MetricDefinition['source'], + trustGate?: TrustGate, + layer?: TrustLayer, +): number { + let files: string[]; + try { + files = fs.readdirSync(dir).filter(f => f.endsWith('.metric.md')); + } catch { + return 0; + } + if (layer === 'project' && !isWorkspaceTrusted()) { + warnCore('MetricLoader', 'Workspace is not trusted; skipping project metrics'); + return 0; + } + files = capFiles(files, dir, 'MetricLoader'); + + let count = 0; + for (const file of files) { + const filePath = path.join(dir, file); + try { + const markdown = fs.readFileSync(filePath, 'utf-8'); + if (trustGate && layer && !trustGate.isAllowed(filePath, markdown)) { + trustGate.onBlocked({ filePath, layer, kind: 'metric', hash: hashContent(markdown), content: markdown }); + continue; + } + const metric = parseMetric(markdown); + if (metric) { + metric.source = source; + metric.sourceFilePath = filePath; + registerMetric(metric); + count++; + } + } catch { + warnCore('MetricLoader', `Failed to read ${filePath}`); + } + } + return count; +} + +const fsp = fs.promises; + +export async function registerAllBuiltinRulesAsync(): Promise { + if (builtinRegistered) return; + builtinRegistered = true; + + const rulesDir = path.join(__dirname, 'rules'); + let files: string[]; + let baseDir = rulesDir; + try { + files = (await fsp.readdir(rulesDir)).filter(f => f.endsWith('.md')); + } catch { + const srcRulesDir = path.join(__dirname, '..', 'src', 'core', 'rules'); + try { + files = (await fsp.readdir(srcRulesDir)).filter(f => f.endsWith('.md')); + baseDir = srcRulesDir; + } catch { + warnCore('RuleLoader', 'Could not find rules directory'); + return; + } + } + + await Promise.all(files.map(async file => { + const id = file.replace(/\.md$/, ''); + try { + const source = await fsp.readFile(path.join(baseDir, file), 'utf-8'); + registerBuiltinRuleSource(id, source); + } catch { + warnCore('RuleLoader', `Failed to read ${path.join(baseDir, file)}`); + } + })); +} + +async function loadRulesFromDirAsync( + dir: string, + register: (id: string, source: string, filePath: string) => void, + trustGate?: TrustGate, + layer: TrustLayer = 'personal', +): Promise { + let files: string[]; + try { + files = (await fsp.readdir(dir)).filter(f => f.endsWith('.md')); + } catch { + return 0; + } + if (layer === 'project' && !isWorkspaceTrusted()) { + warnCore('RuleLoader', 'Workspace is not trusted; skipping project rules'); + return 0; + } + files = capFiles(files, dir, 'RuleLoader'); + + let count = 0; + await Promise.all(files.map(async file => { + const filePath = path.join(dir, file); + try { + const id = file.replace(/\.md$/, ''); + const source = await fsp.readFile(filePath, 'utf-8'); + if (trustGate && !trustGate.isAllowed(filePath, source)) { + trustGate.onBlocked({ filePath, layer, kind: 'rule', hash: hashContent(source), content: source }); + return; + } + register(id, source, filePath); + count++; + } catch { + warnCore('RuleLoader', `Failed to read ${filePath}`); + } + })); + return count; +} + +export async function loadPersonalRulesAsync(trustGate?: TrustGate): Promise { + clearLayerRules('personal'); + return loadRulesFromDirAsync(getPersonalRulesDir(), + (id, source, filePath) => registerPersonalRuleSource(id, source, filePath), + resolveGate(trustGate), 'personal'); +} + +export async function loadProjectRulesAsync(workspaceRoot: string, trustGate?: TrustGate): Promise { + clearLayerRules('project'); + return loadRulesFromDirAsync(getProjectRulesDir(workspaceRoot), + (id, source, filePath) => registerProjectRuleSource(id, source, filePath), + resolveGate(trustGate), 'project'); +} + +export async function loadAllRuleLayersAsync(workspaceRoot?: string, trustGate?: TrustGate): Promise<{ builtin: number; personal: number; project: number }> { + await registerAllBuiltinRulesAsync(); + const [personal, project] = await Promise.all([ + loadPersonalRulesAsync(trustGate), + workspaceRoot ? loadProjectRulesAsync(workspaceRoot, trustGate) : Promise.resolve(0), + ]); + return { + builtin: loadBuiltinRules().length, + personal, + project, + }; +} + +export async function registerAllBuiltinMetricsAsync(): Promise { + if (builtinMetricsRegistered) return 0; + builtinMetricsRegistered = true; + + const dirs = [ + path.join(__dirname, 'metrics'), + path.join(__dirname, '..', 'src', 'core', 'metrics'), + ]; + + for (const dir of dirs) { + const count = await loadMetricsFromDirAsync(dir, 'built-in'); + if (count > 0) return count; + } + return 0; +} + +export async function loadPersonalMetricsAsync(trustGate?: TrustGate): Promise { + return loadMetricsFromDirAsync(path.join(os.homedir(), '.ai-engineer-coach', 'metrics'), 'personal', resolveGate(trustGate), 'personal'); +} + +export async function loadProjectMetricsAsync(workspaceRoot: string, trustGate?: TrustGate): Promise { + return loadMetricsFromDirAsync(path.join(workspaceRoot, '.ai-engineer-coach', 'metrics'), 'project', resolveGate(trustGate), 'project'); +} + +export async function loadAllMetricLayersAsync(workspaceRoot?: string, trustGate?: TrustGate): Promise<{ builtin: number; personal: number; project: number }> { + clearMetrics(); + const builtin = await registerAllBuiltinMetricsAsync(); + const [personal, project] = await Promise.all([ + loadPersonalMetricsAsync(trustGate), + workspaceRoot ? loadProjectMetricsAsync(workspaceRoot, trustGate) : Promise.resolve(0), + ]); + return { builtin, personal, project }; +} + +async function loadMetricsFromDirAsync( + dir: string, + source: MetricDefinition['source'], + trustGate?: TrustGate, + layer?: TrustLayer, +): Promise { + let files: string[]; + try { + files = (await fsp.readdir(dir)).filter(f => f.endsWith('.metric.md')); + } catch { + return 0; + } + if (layer === 'project' && !isWorkspaceTrusted()) { + warnCore('MetricLoader', 'Workspace is not trusted; skipping project metrics'); + return 0; + } + files = capFiles(files, dir, 'MetricLoader'); + + let count = 0; + await Promise.all(files.map(async file => { + const filePath = path.join(dir, file); + try { + const markdown = await fsp.readFile(filePath, 'utf-8'); + if (trustGate && layer && !trustGate.isAllowed(filePath, markdown)) { + trustGate.onBlocked({ filePath, layer, kind: 'metric', hash: hashContent(markdown), content: markdown }); + return; + } + const metric = parseMetric(markdown); + if (metric) { + metric.source = source; + metric.sourceFilePath = filePath; + registerMetric(metric); + count++; + } + } catch { + warnCore('MetricLoader', `Failed to read ${filePath}`); + } + })); + return count; +} \ No newline at end of file diff --git a/packages/ai-coach-analytics/src/rule-parser.ts b/packages/ai-coach-analytics/src/rule-parser.ts new file mode 100644 index 0000000..5b30c62 --- /dev/null +++ b/packages/ai-coach-analytics/src/rule-parser.ts @@ -0,0 +1,509 @@ +/*--------------------------------------------------------------------------------------------- + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the MIT License. See LICENSE in the project root for license information. + *--------------------------------------------------------------------------------------------*/ + +/** + * Rule parser: reads markdown-based rule definitions and produces DetectionRule objects. + * + * Rule file format: + * ```markdown + * --- + * id: lazy-prompting + * name: Lazy Prompting + * group: prompt-quality + * severity: medium + * scope: requests + * requiresIdeContext: false + * version: 1 + * tags: [prompt, quality, short] + * thresholds: + * minChars: 30 + * maxRatio: 0.3 + * minSample: 10 + * --- + * + * # Description + * Detects requests with very short prompts that lack sufficient context. + * + * # When Triggered + * {{count}} requests ({{pct}}) are under {{extra.minChars}} characters. Very short prompts often produce poor results. + * + * # How to Improve + * Provide more context in your prompts: describe the intent, constraints, and expected output format. + * + * # Examples + * "{{message}}" ({{extra.charCount}} chars) + * + * # Detection Logic + * ```detect + * filter: messageLength < thresholds.minChars AND messageLength > 0 + * check: ratio > thresholds.maxRatio AND count > thresholds.minSample + * ``` + * ``` + */ + +import { DetectionRule, RuleCondition, RuleSeverity, RuleScope, PracticeGroup, PRACTICE_GROUPS } from './types'; + +interface ParsedFrontmatter { + id: string; + name: string; + group: string; + severity: string; + scope?: string; + requiresIdeContext?: boolean; + version?: number; + tags?: string[]; + thresholds?: Record; + patterns?: Record; + fileTypes?: Record; + extends?: string; + /** Referenced metric ID (from .metric.md files) */ + metric?: string; + /** Rule dependency IDs */ + requires?: string[]; +} + +function parseNestedValue(line: string, nestedObj: Record): void { + const colonIdx = line.indexOf(':'); + const k = line.substring(0, colonIdx).trim(); + const v = line.substring(colonIdx + 1).trim(); + if (v.startsWith('[') && v.endsWith(']')) { + nestedObj[k] = v.slice(1, -1).split(',').map(s => s.trim().replaceAll(/^['"]|['"]$/g, '')); + return; + } + const num = Number.parseFloat(v); + if (!Number.isNaN(num) && String(num) === v) { + nestedObj[k] = num; + return; + } + nestedObj[k] = v.replaceAll(/^['"]|['"]$/g, ''); +} + +function parseTopLevelValue(value: string): unknown { + if (value.startsWith('[') && value.endsWith(']')) { + return value.slice(1, -1).split(',').map(s => s.trim().replaceAll(/^['"]|['"]$/g, '')); + } + if (value === 'true') return true; + if (value === 'false') return false; + const num = Number.parseFloat(value); + if (!Number.isNaN(num) && String(num) === value) return num; + return value.replaceAll(/^['"]|['"]$/g, ''); +} + +function parseFrontmatter(raw: string): { frontmatter: ParsedFrontmatter; body: string } | null { + const match = raw.match(/^---\s*\n([\s\S]*?)\n---\s*\n([\s\S]*)$/); + if (!match) return null; + + const yamlBlock = match[1]; + const body = match[2]; + + const fm: Record = {}; + let currentKey = ''; + let inNestedBlock = false; + const nestedObj: Record = {}; + + for (const line of yamlBlock.split('\n')) { + const trimmed = line.trim(); + if (!trimmed || trimmed.startsWith('#')) continue; + + if (inNestedBlock) { + if (/^\s{2,}/.test(line) && line.includes(':')) { + parseNestedValue(line, nestedObj); + continue; + } + fm[currentKey] = { ...nestedObj }; + inNestedBlock = false; + } + + const kvMatch = trimmed.match(/^(\w[\w-]*):\s*(.*)$/); + if (!kvMatch) continue; + + const [, key, rawValue] = kvMatch; + const value = rawValue.trim(); + + if (value === '' || value === '|') { + currentKey = key; + inNestedBlock = true; + for (const k of Object.keys(nestedObj)) delete nestedObj[k]; + continue; + } + + fm[key] = parseTopLevelValue(value); + } + + if (inNestedBlock) { + fm[currentKey] = { ...nestedObj }; + } + + return { frontmatter: fm as unknown as ParsedFrontmatter, body }; +} + +function parseSection(body: string, heading: string): string { + // Find the heading, then capture everything until the next heading or true end-of-string. + // We avoid the 'm' flag on `$` which would stop at end-of-line; instead we use + // a negative lookahead `(?![\s\S])` as a true end-of-string anchor. + const regex = new RegExp(`^#+\\s+${heading}\\s*\\n([\\s\\S]*?)(?=^#+\\s|(?![\\s\\S]))`, 'mi'); + const match = body.match(regex); + return match ? match[1].trim() : ''; +} + +function parseDetectionLogic(body: string): RuleCondition[] { + const logicBlock = body.match(/```detect\s*\n([\s\S]*?)```/); + if (!logicBlock) return []; + + const lines = logicBlock[1].split('\n').filter(l => l.trim()); + const conditions: RuleCondition[] = []; + + for (const line of lines) { + const trimmed = line.trim(); + + if (trimmed.startsWith('filter:')) { + conditions.push(parseFilterLine(trimmed.slice(7).trim())); + } else if (trimmed.startsWith('check:')) { + conditions.push(parseCheckLine(trimmed.slice(6).trim())); + } else if (trimmed.startsWith('aggregate:')) { + conditions.push(parseAggregateLine(trimmed.slice(10).trim())); + } else if (trimmed.startsWith('group-by:')) { + conditions.push({ type: 'group-by', field: trimmed.slice(9).trim() }); + } else if (trimmed.startsWith('match:')) { + const pattern = trimmed.slice(6).trim(); + conditions.push({ type: 'regex-match', pattern, flags: 'i' }); + } else if (trimmed.startsWith('custom:')) { + conditions.push({ type: 'custom', field: trimmed.slice(7).trim() }); + } + } + + return conditions; +} + +function parseFilterLine(expr: string): RuleCondition { + // Parse expressions like: messageLength < thresholds.minChars AND messageLength > 0 + const parts = expr.split(/\s+AND\s+/i); + if (parts.length > 1) { + return { + type: 'filter-requests', + logic: 'and', + conditions: parts.map(p => parseSingleCondition(p.trim())), + }; + } + const orParts = expr.split(/\s+OR\s+/i); + if (orParts.length > 1) { + return { + type: 'filter-requests', + logic: 'or', + conditions: orParts.map(p => parseSingleCondition(p.trim())), + }; + } + return parseSingleCondition(expr); +} + +function parseCheckLine(expr: string): RuleCondition { + const parts = expr.split(/\s+AND\s+/i); + const conditions = parts.map(p => { + const trimmed = p.trim(); + if (trimmed.startsWith('ratio')) { + const match = trimmed.match(/ratio\s*(>|<|>=|<=)\s*(.+)/); + if (match) { + return { + type: 'ratio-check' as const, + op: parseOp(match[1]), + threshold: parseValue(match[2]) as number, + }; + } + } + if (trimmed.startsWith('count')) { + const match = trimmed.match(/count\s*(>|<|>=|<=)\s*(.+)/); + if (match) { + return { + type: 'count-check' as const, + op: parseOp(match[1]), + value: parseValue(match[2]), + }; + } + } + return parseSingleCondition(trimmed); + }); + + if (conditions.length === 1) return conditions[0]; + return { type: 'filter-requests', logic: 'and', conditions: conditions }; +} + +function parseSingleCondition(expr: string): RuleCondition { + const match = expr.match(/^(\S+)\s*(<=|>=|!=|<|>|==|=|includes|not-includes|matches|not-matches)\s*(.+)$/); + if (!match) return { type: 'field-check', field: expr }; + + const [, field, rawOp, rawVal] = match; + return { + type: 'field-check', + field, + op: parseOp(rawOp), + value: parseValue(rawVal), + }; +} + +function parseOp(op: string): RuleCondition['op'] { + switch (op) { + case '<': return 'lt'; + case '>': return 'gt'; + case '<=': return 'lte'; + case '>=': return 'gte'; + case '==': case '=': return 'eq'; + case '!=': return 'ne'; + default: return op as RuleCondition['op']; + } +} + +function parseValue(val: string): string | number | boolean { + const trimmed = val.trim(); + if (trimmed === 'true') return true; + if (trimmed === 'false') return false; + const num = Number.parseFloat(trimmed); + if (!Number.isNaN(num) && /^[\d.]+$/.test(trimmed)) return num; + // References like thresholds.xxx stay as strings + return trimmed; +} + +function parseAggregateLine(expr: string): RuleCondition { + const match = expr.match(/^(\w+)\((\S+)\)\s*(>|<|>=|<=)\s*(.+)$/); + if (match) { + return { + type: 'aggregate', + aggregation: match[1] as RuleCondition['aggregation'], + field: match[2], + op: parseOp(match[3]), + value: parseValue(match[4]), + }; + } + return { type: 'aggregate', field: expr }; +} + +function extractRuleMaps(fm: ParsedFrontmatter): { + thresholds: Record; + patterns: Record; + fileTypes: Record; +} { + const rawThresholds = fm.thresholds as Record | undefined; + const thresholds: Record = {}; + if (rawThresholds) { + for (const [k, v] of Object.entries(rawThresholds)) { + if (typeof v === 'number') thresholds[k] = v; + } + } + + const patterns: Record = {}; + const rawPatterns = fm.patterns as Record | undefined; + if (rawPatterns) { + for (const [k, v] of Object.entries(rawPatterns)) { + if (Array.isArray(v)) patterns[k] = v; + else if (typeof v === 'string') patterns[k] = [v]; + } + } + + const fileTypes: Record = {}; + const rawFileTypes = fm.fileTypes as Record | undefined; + if (rawFileTypes) { + for (const [k, v] of Object.entries(rawFileTypes)) { + if (Array.isArray(v)) fileTypes[k] = v; + else if (typeof v === 'string') fileTypes[k] = [v]; + } + } + + return { thresholds, patterns, fileTypes }; +} + +const VALID_GROUPS = new Set(Object.keys(PRACTICE_GROUPS)); +const VALID_SEVERITIES = new Set(['high', 'medium', 'low']); +const VALID_SCOPES = new Set(['requests', 'sessions', 'both']); + +export function parseRule(markdown: string): DetectionRule | null { + const parsed = parseFrontmatter(markdown); + if (!parsed) return null; + + const { frontmatter: fm, body } = parsed; + + if (!fm.id || !fm.name || !fm.group || !fm.severity) return null; + if (!VALID_GROUPS.has(fm.group)) return null; + if (!VALID_SEVERITIES.has(fm.severity as RuleSeverity)) return null; + + const scope = (fm.scope || 'requests') as RuleScope; + if (!VALID_SCOPES.has(scope)) return null; + + const description = parseSection(body, 'Description'); + const descriptionTemplate = parseSection(body, 'When Triggered'); + const suggestionTemplate = parseSection(body, 'How to Improve'); + const exampleTemplate = parseSection(body, 'Examples'); + const conditions = parseDetectionLogic(body); + const tests = parseTestFixtures(body); + + const { thresholds, patterns, fileTypes } = extractRuleMaps(fm); + + return { + id: fm.id, + name: fm.name, + group: fm.group as PracticeGroup, + severity: fm.severity as RuleSeverity, + requiresIdeContext: fm.requiresIdeContext ?? false, + scope, + description: description || `Detects ${fm.name.toLowerCase()} patterns.`, + descriptionTemplate: descriptionTemplate || '{{count}} occurrences detected.', + suggestionTemplate: suggestionTemplate || 'Review your practices.', + exampleTemplate: exampleTemplate || '', + maxExamples: 3, + conditions, + thresholds, + patterns, + fileTypes, + extendsRule: fm.extends ? String(fm.extends) : undefined, + tests, + source: 'built-in', + sourceFilePath: '', + version: fm.version || 1, + tags: fm.tags || [], + rawSource: markdown, + }; +} + +/** + * Parse # Tests section from rule body. + * Format: ```test blocks with JSON-like lines: {key: val, ...} -> triggered|clean + */ +function parseTestFixtures(body: string): Array<{ input: Record; expect: 'triggered' | 'clean' }> { + const block = body.match(/```test\s*\n([\s\S]*?)```/); + if (!block) return []; + const tests: Array<{ input: Record; expect: 'triggered' | 'clean' }> = []; + for (const line of block[1].split('\n')) { + const trimmed = line.trim(); + if (!trimmed) continue; + const arrowIdx = trimmed.lastIndexOf('->'); + if (arrowIdx < 0) continue; + const jsonPart = trimmed.substring(0, arrowIdx).trim(); + const expectPart = trimmed.substring(arrowIdx + 2).trim().toLowerCase(); + if (expectPart !== 'triggered' && expectPart !== 'clean') continue; + try { + // Convert relaxed JSON to strict: add quotes to keys + const strict = jsonPart.replaceAll(/(\w+)\s*:/g, '"$1":').replaceAll('\'', '"'); + const input = JSON.parse(strict) as Record; + tests.push({ input, expect: expectPart }); + } catch { /* skip unparseable lines */ } + } + return tests; +} + +/** + * Serialize a DetectionRule back to markdown format. + */ +export function serializeRule(rule: DetectionRule): string { + const lines: string[] = ['---']; + lines.push(`id: ${rule.id}`); + lines.push(`name: ${rule.name}`); + lines.push(`group: ${rule.group}`); + lines.push(`severity: ${rule.severity}`); + lines.push(`scope: ${rule.scope}`); + if (rule.requiresIdeContext) lines.push(`requiresIdeContext: true`); + lines.push(`version: ${rule.version}`); + if (rule.tags.length > 0) lines.push(`tags: [${rule.tags.join(', ')}]`); + if (rule.extendsRule) lines.push(`extends: ${rule.extendsRule}`); + if (Object.keys(rule.thresholds).length > 0) { + lines.push('thresholds:'); + for (const [k, v] of Object.entries(rule.thresholds)) { + lines.push(` ${k}: ${v}`); + } + } + if (Object.keys(rule.patterns).length > 0) { + lines.push('patterns:'); + for (const [k, v] of Object.entries(rule.patterns)) { + lines.push(` ${k}: [${v.map(s => `"${s}"`).join(', ')}]`); + } + } + if (Object.keys(rule.fileTypes).length > 0) { + lines.push('fileTypes:'); + for (const [k, v] of Object.entries(rule.fileTypes)) { + lines.push(` ${k}: [${v.join(', ')}]`); + } + } + lines.push('---'); + lines.push(''); + lines.push(`# Description`); + lines.push(rule.description); + lines.push(''); + lines.push(`# When Triggered`); + lines.push(rule.descriptionTemplate); + lines.push(''); + lines.push(`# How to Improve`); + lines.push(rule.suggestionTemplate); + lines.push(''); + if (rule.exampleTemplate) { + lines.push(`# Examples`); + lines.push(rule.exampleTemplate); + lines.push(''); + } + if (rule.conditions.length > 0) { + lines.push(`# Detection Logic`); + lines.push('```detect'); + for (const c of rule.conditions) { + lines.push(serializeCondition(c)); + } + lines.push('```'); + lines.push(''); + } + return lines.join('\n'); +} + +function serializeCondition(c: RuleCondition): string { + if (c.type === 'custom') return `custom: ${c.field || ''}`; + if (c.type === 'filter-requests' || c.type === 'filter-sessions') { + if (c.conditions && c.conditions.length > 0) { + const joiner = c.logic === 'or' ? ' OR ' : ' AND '; + return `filter: ${c.conditions.map(serializeSubCondition).join(joiner)}`; + } + return `filter: ${serializeSubCondition(c)}`; + } + if (c.type === 'ratio-check') return `check: ratio ${opToStr(c.op)} ${c.threshold}`; + if (c.type === 'count-check') return `check: count ${opToStr(c.op)} ${c.value}`; + if (c.type === 'aggregate') return `aggregate: ${c.aggregation || 'count'}(${c.field || ''}) ${opToStr(c.op)} ${c.value}`; + if (c.type === 'regex-match') return `match: ${c.pattern}`; + if (c.type === 'group-by') return `group-by: ${c.field}`; + return `filter: ${serializeSubCondition(c)}`; +} + +function serializeSubCondition(c: RuleCondition): string { + if (!c.field) return ''; + return `${c.field} ${opToStr(c.op)} ${c.value ?? ''}`; +} + +function opToStr(op: RuleCondition['op']): string { + switch (op) { + case 'lt': return '<'; + case 'gt': return '>'; + case 'lte': return '<='; + case 'gte': return '>='; + case 'eq': return '=='; + case 'ne': return '!='; + default: return op || '=='; + } +} + +/** + * Fill a template string with variables. + * Supports {{var}}, {{extra.var}}, and simple number formatting. + */ +export function fillTemplate(template: string, vars: Record): string { + return template.replaceAll(/\{\{(\w+(?:\.\w+)*)\}\}/g, (_match, path: string) => { + const parts = path.split('.'); + let val: unknown = vars; + for (const p of parts) { + if (val && typeof val === 'object') val = (val as Record)[p]; + else return ''; + } + if (typeof val === 'number') { + if (Number.isInteger(val)) return String(val); + return val.toFixed(1); + } + if (val == null) return ''; + if (typeof val === 'string') return val; + if (typeof val === 'boolean') return String(val); + if (Array.isArray(val)) return val.join(', '); + return JSON.stringify(val); + }); +} diff --git a/packages/ai-coach-analytics/src/rule-pipeline.ts b/packages/ai-coach-analytics/src/rule-pipeline.ts new file mode 100644 index 0000000..2ddfd4e --- /dev/null +++ b/packages/ai-coach-analytics/src/rule-pipeline.ts @@ -0,0 +1,366 @@ +/*--------------------------------------------------------------------------------------------- + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the MIT License. See LICENSE in the project root for license information. + *--------------------------------------------------------------------------------------------*/ + +/** + * Universal rule pipeline engine. + * + * Compiles a Detection Logic block from a rule .md file into an emitter+trigger + * pair using declarative DSL directives (`scan:`, `match:`, `aggregate:`, + * `check:`, `examples:`, `severity:`). + * + * Also handles: + * - **Inheritance** (`extends:` in frontmatter) + * - **Test fixture validation** (`# Tests` section) + * - **Pattern/fileType resolution** into the DSL evaluation context + */ + +import type { DetectionRule, DetectorEmission, RuleSeverity, Session, SessionRequest } from './types'; +import { compileFilter, compileTrigger, evaluateTemplate, evaluateExpression } from './dsl/index'; +import { getAllRules } from './rule-engine'; +import { warnCore } from './log'; + +/* ================================================================== */ +/* Pipeline Definition */ +/* ================================================================== */ + +export interface Pipeline { + scan: 'requests' | 'sessions'; + matchExpr?: string; + aggregate: 'count' | 'ratio'; + checkExpr?: string; + examplesTemplate?: string; + severityExpr?: string; + reduceExprs?: Record; +} + +export interface DetectorContext { + reqs: SessionRequest[]; + sessions: Session[]; + skipIdeDetectors: boolean; +} + +/* ================================================================== */ +/* Pipeline Parser */ +/* ================================================================== */ + +/** + * Parse the Detection Logic block from a rule's raw markdown source. + * Returns a Pipeline definition. + */ +function mergeLineContinuations(rawLines: string[]): string[] { + const lines: string[] = []; + for (let i = 0; i < rawLines.length; i++) { + let line = rawLines[i]; + while (line.endsWith('\\') && i + 1 < rawLines.length) { + line = line.slice(0, -1).trimEnd() + ' ' + rawLines[++i]; + } + lines.push(line); + } + return lines; +} + +export function parsePipeline(rule: DetectionRule): Pipeline { + const block = rule.rawSource.match(/```detect\s*\n([\s\S]*?)```/); + if (!block) { + // No detection logic → default pipeline that never triggers + return { scan: rule.scope === 'sessions' ? 'sessions' : 'requests', aggregate: 'count' }; + } + + const rawLines = block[1].split('\n').map(l => l.trim()).filter(Boolean); + const lines = mergeLineContinuations(rawLines); + const pipeline: Pipeline = { + scan: rule.scope === 'sessions' ? 'sessions' : 'requests', + aggregate: 'count', + }; + + for (const line of lines) { + const colonIdx = line.indexOf(':'); + if (colonIdx < 0) continue; + const rawKey = line.substring(0, colonIdx).trim(); + const key = rawKey.toLowerCase(); + const value = line.substring(colonIdx + 1).trim(); + + switch (key) { + case 'scan': + pipeline.scan = value === 'sessions' ? 'sessions' : 'requests'; + break; + case 'match': + pipeline.matchExpr = value; + break; + case 'aggregate': + pipeline.aggregate = value === 'ratio' ? 'ratio' : 'count'; + break; + case 'check': + pipeline.checkExpr = value; + break; + case 'examples': + pipeline.examplesTemplate = value; + break; + case 'severity': + pipeline.severityExpr = value; + break; + default: + // Any other key:value is treated as a reduce expression (preserve original case) + if (rawKey && value && key !== 'reduce') { + if (!pipeline.reduceExprs) pipeline.reduceExprs = {}; + pipeline.reduceExprs[rawKey] = value; + } + break; + } + } + + return pipeline; +} + +/* ================================================================== */ +/* Rule Context Builder */ +/* ================================================================== */ + +/** + * Build the evaluation context that gets merged into each row when + * the DSL filter runs. This injects thresholds, patterns, and fileTypes + * from the rule's frontmatter so expressions can reference them. + */ +function buildRuleContext(rule: DetectionRule): Record { + const ctx: Record = {}; + + // thresholds.xxx → thresholds object + if (rule.thresholds) ctx.thresholds = rule.thresholds; + // patterns.xxx → patterns object + if (rule.patterns && Object.keys(rule.patterns).length > 0) ctx.patterns = rule.patterns; + // fileTypes.xxx → fileTypes object + if (rule.fileTypes && Object.keys(rule.fileTypes).length > 0) ctx.fileTypes = rule.fileTypes; + + return ctx; +} + +/* ================================================================== */ +/* Pipeline Executor */ +/* ================================================================== */ + +/** + * Execute a pipeline against data to produce a DetectorEmission. + */ +function buildExamples( + matched: Record[], + pipeline: Pipeline, + ruleCtx: Record, + count: number, + total: number, + ratio: number, + extra: Record, +): string[] { + const examples: string[] = []; + for (const row of matched.slice(0, 3)) { + if (pipeline.examplesTemplate) { + const mergedRow = { ...row, ...ruleCtx, count, total, ratio, ...extra }; + examples.push(evaluateTemplate(pipeline.examplesTemplate, mergedRow)); + } else { + const msgText = row.messageText; + const wsName = row.workspaceName; + const msg = typeof msgText === 'string' ? msgText : (typeof wsName === 'string' ? wsName : ''); + if (msg) examples.push(msg.substring(0, 80)); + } + } + return examples; +} + +export function executePipeline( + pipeline: Pipeline, + rule: DetectionRule, + ctx: DetectorContext, +): DetectorEmission { + // ── Declarative pipeline path ── + const rows: Record[] = pipeline.scan === 'sessions' + ? ctx.sessions as unknown as Record[] + : ctx.reqs as unknown as Record[]; + + if (rows.length === 0) return EMPTY_EMISSION; + + const ruleCtx = buildRuleContext(rule); + + // Compile match filter + const matchFn = pipeline.matchExpr + ? compileFilterWithContext(pipeline.matchExpr, ruleCtx) + : () => true; + + // Filter rows + const matched = rows.filter(matchFn); + + // Aggregate + let count = matched.length; + let total = rows.length; + let ratio = total > 0 ? count / total : 0; + + // Evaluate reduce expressions to compute extra fields + const extra: Record = {}; + if (pipeline.reduceExprs) { + const reduceCtx: Record = { + ...ruleCtx, + matched, + all: rows, + allReqs: ctx.reqs, + allSessions: ctx.sessions, + count, + total, + ratio, + extra, + }; + for (const [key, expr] of Object.entries(pipeline.reduceExprs)) { + const val = evaluateExpression(expr, reduceCtx); + extra[key] = val; + // Make each computed value available to subsequent expressions + reduceCtx[key] = val; + reduceCtx.extra = { ...extra }; + } + } + + // Override count/total/ratio from reduce expressions (emitCount, emitTotal, emitRatio) + if (typeof extra.emitCount === 'number') count = extra.emitCount; + if (typeof extra.emitTotal === 'number') total = extra.emitTotal; + if (extra.emitCount !== undefined || extra.emitTotal !== undefined) { + ratio = total > 0 ? count / total : 0; + } + if (typeof extra.emitRatio === 'number') ratio = extra.emitRatio; + + // Generate examples + const examples = buildExamples(matched, pipeline, ruleCtx, count, total, ratio, extra); + + // Dynamic severity + let dynamicSeverity: RuleSeverity | undefined; + if (pipeline.severityExpr) { + try { + const severityFn = compileTrigger(pipeline.severityExpr); + if (severityFn({ count, total, ratio, extra })) { + dynamicSeverity = 'high'; + } + } catch { /* ignore severity expression errors */ } + } + + return { count, total, ratio, examples, extra, dynamicSeverity, matchedRows: matched }; +} + +/** + * Check whether a pipeline's trigger condition is met. + */ +export function checkPipelineTrigger( + pipeline: Pipeline, + emission: DetectorEmission, + rule: DetectionRule, +): boolean { + if (!pipeline.checkExpr) { + // Default: trigger if any matches + return emission.count > 0; + } + + // Resolve threshold references in the check expression + let resolved = pipeline.checkExpr; + if (rule.thresholds) { + for (const [k, v] of Object.entries(rule.thresholds)) { + resolved = resolved.replaceAll(`thresholds.${k}`, String(v)); + } + } + + try { + const triggerFn = compileTrigger(resolved); + return triggerFn({ + count: emission.count, + total: emission.total, + ratio: emission.ratio, + extra: emission.extra, + }); + } catch { + return emission.count > 0; + } +} + +/* ================================================================== */ +/* Inheritance Resolver */ +/* ================================================================== */ + +/** + * Resolve rule inheritance. If a rule has `extends: parent-id`, merge + * the parent's fields (thresholds, patterns, fileTypes, templates) with + * the child's overrides. + */ +export function resolveInheritance(rule: DetectionRule, visited: Set = new Set()): DetectionRule { + if (!rule.extendsRule) return rule; + + if (visited.has(rule.id)) { + warnCore('Pipeline', `Circular inheritance detected: ${[...visited].join(' -> ')} -> ${rule.id}`); + return rule; + } + visited.add(rule.id); + + const allRules = getAllRules(); + const parent = allRules.find(r => r.id === rule.extendsRule); + if (!parent) { + warnCore('Pipeline', `Inheritance: parent rule '${rule.extendsRule}' not found for '${rule.id}'`); + return rule; + } + + // Resolve parent first (chains supported) + const resolvedParent = resolveInheritance(parent, visited); + + return { + ...resolvedParent, + // Child overrides: + id: rule.id, + name: rule.name || resolvedParent.name, + severity: rule.severity || resolvedParent.severity, + group: rule.group || resolvedParent.group, + scope: rule.scope || resolvedParent.scope, + description: rule.description || resolvedParent.description, + descriptionTemplate: rule.descriptionTemplate || resolvedParent.descriptionTemplate, + suggestionTemplate: rule.suggestionTemplate || resolvedParent.suggestionTemplate, + exampleTemplate: rule.exampleTemplate || resolvedParent.exampleTemplate, + // Merge thresholds (child wins) + thresholds: { ...resolvedParent.thresholds, ...rule.thresholds }, + // Merge patterns (child wins per key) + patterns: { ...resolvedParent.patterns, ...rule.patterns }, + // Merge fileTypes (child wins per key) + fileTypes: { ...resolvedParent.fileTypes, ...rule.fileTypes }, + // Keep child's detection logic if present + rawSource: rule.rawSource, + conditions: rule.conditions.length > 0 ? rule.conditions : resolvedParent.conditions, + tags: rule.tags.length > 0 ? rule.tags : resolvedParent.tags, + version: rule.version || resolvedParent.version, + source: rule.source, + sourceFilePath: rule.sourceFilePath, + tests: rule.tests.length > 0 ? rule.tests : resolvedParent.tests, + }; +} + +/* ================================================================== */ +/* Helpers */ +/* ================================================================== */ + +const EMPTY_EMISSION: DetectorEmission = { + count: 0, total: 0, ratio: 0, examples: [], extra: {}, +}; + +/** + * Compile a DSL filter expression with rule-level context merged into + * each row's evaluation scope. + */ +function compileFilterWithContext( + expr: string, + ruleCtx: Record, +): (row: Record) => boolean { + // Resolve thresholds.xxx references to literal values + let resolved = expr; + const thresholds = ruleCtx.thresholds as Record | undefined; + if (thresholds) { + for (const [k, v] of Object.entries(thresholds)) { + resolved = resolved.replaceAll(`thresholds.${k}`, String(v)); + } + } + + const filterFn = compileFilter(resolved); + return (row: Record) => { + const mergedRow = { ...row, ...ruleCtx }; + return filterFn(mergedRow); + }; +} diff --git a/packages/ai-coach-analytics/src/rule-trust.ts b/packages/ai-coach-analytics/src/rule-trust.ts new file mode 100644 index 0000000..6d58829 --- /dev/null +++ b/packages/ai-coach-analytics/src/rule-trust.ts @@ -0,0 +1,177 @@ +/*--------------------------------------------------------------------------------------------- + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the MIT License. See LICENSE in the project root for license information. + *--------------------------------------------------------------------------------------------*/ + +/** + * Trust gate for locally-authored rule / metric markdown files. + * + * Built-in rules ship with the extension and are implicitly trusted. + * Personal (`~/.ai-engineer-coach/rules|metrics/`) and project + * (`/.ai-engineer-coach/rules|metrics/`) files are NOT trusted by + * default: a malicious repository could drop a `.ai-engineer-coach/rules/` + * directory whose DSL executes the moment the dashboard is opened. + * + * This module implements a "trust on first use" policy: + * - Each local file's content is hashed (SHA-256) the first time we see it. + * - The hash + absolute path is recorded in the extension's `globalState` + * once the user explicitly approves it. + * - On subsequent loads, the recorded hash must match the current file + * contents exactly; any edit invalidates prior approval. + * - Files that fail the check are skipped by the loader and queued in an + * in-memory pending list for the UI to surface. + * + * This module is pure: it accepts a `Memento`-compatible storage object (so + * tests can inject a Map-backed fake) and has no dependency on `vscode`. + */ + +import { createHash } from 'crypto'; +import * as path from 'path'; + +/** Minimum surface of `vscode.Memento` that we need. */ +export interface TrustMemento { + get(key: string, defaultValue: T): T; + update(key: string, value: unknown): Promise; +} + +export interface ApprovedEntry { + hash: string; + approvedAt: number; +} + +export type ApprovalMap = Record; + +/** Key used in `globalState` to persist the approval map. */ +const STORAGE_KEY = 'aiEngineerCoach.ruleTrust.v1'; + +/** Compute a stable content hash. */ +export function hashContent(content: string): string { + return createHash('sha256').update(content, 'utf8').digest('hex'); +} + +/** + * Canonical storage key for an approval. Resolves to an absolute path and + * case-folds on Windows so the same file cannot carry two different + * approvals via path-case aliases on a case-insensitive filesystem. + * (Content hashing remains the actual security gate; this removes key + * ambiguity.) + */ +export function canonicalApprovalKey(filePath: string): string { + const resolved = path.resolve(filePath); + return process.platform === 'win32' ? resolved.toLowerCase() : resolved; +} + +/** Read the approval map from storage. */ +export function listApproved(store: TrustMemento): ApprovalMap { + return { ...store.get(STORAGE_KEY, {}) }; +} + +/** True iff this exact file + content has been previously approved. */ +export function isApproved(store: TrustMemento, filePath: string, content: string): boolean { + const approvals = store.get(STORAGE_KEY, {}); + // Legacy fallback: approvals recorded before keys were canonicalized. + const entry = approvals[canonicalApprovalKey(filePath)] ?? approvals[filePath]; + if (!entry) return false; + return entry.hash === hashContent(content); +} + +/** Record approval for a file at its current content. */ +export async function approve(store: TrustMemento, filePath: string, content: string): Promise { + const approvals: ApprovalMap = { ...store.get(STORAGE_KEY, {}) }; + approvals[canonicalApprovalKey(filePath)] = { hash: hashContent(content), approvedAt: Date.now() }; + await store.update(STORAGE_KEY, approvals); +} + +/** Revoke approval for a single file. */ +export async function revoke(store: TrustMemento, filePath: string): Promise { + const approvals: ApprovalMap = { ...store.get(STORAGE_KEY, {}) }; + // Delete the canonical key plus any legacy raw-path entry. + const keys = [canonicalApprovalKey(filePath), filePath].filter(k => k in approvals); + if (keys.length > 0) { + for (const k of keys) delete approvals[k]; + await store.update(STORAGE_KEY, approvals); + } +} + +/** Remove every approval (for diagnostics / user reset). */ +export async function clearAllApprovals(store: TrustMemento): Promise { + await store.update(STORAGE_KEY, {}); +} + +/* ============================================================== */ +/* Pending list (in-memory) + TrustGate interface */ +/* ============================================================== */ + +export type TrustLayer = 'personal' | 'project'; +export type TrustKind = 'rule' | 'metric'; + +export interface PendingEntry { + filePath: string; + layer: TrustLayer; + kind: TrustKind; + hash: string; + /** Raw file contents -- retained so the UI can show a preview and the + * loader can re-register the file after approval without re-reading it. */ + content: string; +} + +const pendingByPath = new Map(); + +/** Add or replace a pending entry. Keyed by absolute path. */ +export function addPending(entry: PendingEntry): void { + pendingByPath.set(entry.filePath, entry); +} + +/** Remove a single pending entry (e.g. after approval). */ +export function removePending(filePath: string): void { + pendingByPath.delete(filePath); +} + +/** Snapshot of currently pending entries. */ +export function getPending(): PendingEntry[] { + return Array.from(pendingByPath.values()); +} + +/** Clear the pending list (e.g. when reloading all layers). */ +export function clearPending(): void { + pendingByPath.clear(); +} + +/** + * The object loaders consult for every personal / project markdown file. + * When no gate is supplied, loaders fall back to "allow everything" -- + * preserving the long-standing behaviour relied upon by tests and scripts. + */ +export interface TrustGate { + /** Return true to admit the file, false to block it. */ + isAllowed(filePath: string, content: string): boolean; + /** Called with full context for every blocked file. */ + onBlocked(entry: PendingEntry): void; +} + +/** + * Build a `TrustGate` that checks approvals in the given Memento store and + * records every blocked file in the shared pending list. + */ +export function createTrustGate(store: TrustMemento): TrustGate { + return { + isAllowed(filePath, content) { + return isApproved(store, filePath, content); + }, + onBlocked(entry) { + addPending(entry); + }, + }; +} + +/* ============================================================== */ +/* Default store accessor (used by helpers that lack a context) */ +/* ============================================================== */ + +let defaultStore: TrustMemento | undefined; +export function setDefaultTrustStore(store: TrustMemento | undefined): void { + defaultStore = store; +} +export function getDefaultTrustStore(): TrustMemento | undefined { + return defaultStore; +} diff --git a/packages/ai-coach-analytics/src/rules/abandon-sessions.md b/packages/ai-coach-analytics/src/rules/abandon-sessions.md new file mode 100644 index 0000000..88f35c7 --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/abandon-sessions.md @@ -0,0 +1,33 @@ +--- +id: abandon-sessions +name: Abandoned Sessions +group: session-hygiene +severity: low +scope: sessions +version: 1 +tags: [session, abandoned, single] +thresholds: + maxAbandonRate: 0.4 + minSample: 10 +--- + +# Description +Detects sessions with only a single message, indicating missed refinement opportunities. + +# When Triggered +{{count}} sessions ({{pct}}) have only 1 message. You may be missing refinement opportunities. + +# How to Improve +Use follow-up messages to refine Copilot's responses. Iterating produces much better results than one-shot prompts. + +# Examples +{{extra.workspace}}: "{{message}}..." + +# Detection Logic +```detect +scan: sessions +match: requestCount == 1 +aggregate: ratio +check: ratio > thresholds.maxAbandonRate AND count > thresholds.minSample +examples: {{workspaceName}}: abandoned after 1 message +``` diff --git a/packages/ai-coach-analytics/src/rules/agent-mode-for-asks.md b/packages/ai-coach-analytics/src/rules/agent-mode-for-asks.md new file mode 100644 index 0000000..1c4e993 --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/agent-mode-for-asks.md @@ -0,0 +1,42 @@ +--- +id: agent-mode-for-asks +name: Agent Mode for Simple Questions +group: prompt-quality +severity: medium +scope: requests +requiresIdeContext: true +version: 1 +tags: [agent, mode, routing, cost] +thresholds: + maxMessageLength: 80 + minSample: 10 + maxRatio: 0.15 +--- + +# Description +Detects agent-mode requests with very short prompts that didn't trigger any tools, edits, or file context. These look like simple questions that would be cheaper and faster in Ask/Chat mode without the agent loop overhead. + +# When Triggered +{{count}} agent-mode requests ({{pct}}) were trivially short ({{extra.maxMessageLength}} chars) with no tool calls, file edits, or code output. Routing simple questions through agent mode pays for the agent loop without using its capabilities. + +# How to Improve +Use Ask/Chat mode for quick questions ("what does this error mean?", "how do I do X?"). Reserve agent mode for tasks that need to run terminal commands, edit files, or coordinate multiple steps. + +# Examples +"{{messageText | truncate:60}}" ({{messageLength}} chars, agent mode) + +# Detection Logic +```detect +scan: requests +match: agentMode == "agent" AND messageLength > 0 AND messageLength < thresholds.maxMessageLength AND length(toolsUsed) == 0 AND length(aiCode) == 0 AND length(referencedFiles) == 0 AND length(editedFiles) == 0 AND isCanceled == false +aggregate: ratio +check: ratio > thresholds.maxRatio AND count > thresholds.minSample +examples: "{{messageText | truncate:60}}" ({{messageLength}} chars) +``` + +# Tests +```test +{agentMode: "agent", messageText: "what is jwt?", messageLength: 12, toolsUsed: [], aiCode: [], referencedFiles: [], editedFiles: [], isCanceled: false} -> triggered +{agentMode: "agent", messageText: "fix bug", messageLength: 7, toolsUsed: ["edit_file"], aiCode: [], referencedFiles: [], editedFiles: [], isCanceled: false} -> clean +{agentMode: "chat", messageText: "what is jwt?", messageLength: 12, toolsUsed: [], aiCode: [], referencedFiles: [], editedFiles: [], isCanceled: false} -> clean +``` diff --git a/packages/ai-coach-analytics/src/rules/agentic-no-tools.md b/packages/ai-coach-analytics/src/rules/agentic-no-tools.md new file mode 100644 index 0000000..840dc74 --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/agentic-no-tools.md @@ -0,0 +1,33 @@ +--- +id: agentic-no-tools +name: Agentic Without Tools +group: tool-mastery +severity: low +scope: requests +requiresIdeContext: true +version: 1 +tags: [tools, agent, effectiveness] +thresholds: + minSample: 10 +--- + +# Description +Detects agentic requests that used no tools, reducing the effectiveness of agent mode. + +# When Triggered +{{count}} agentic requests used no tools. Agent mode is most effective when tools are enabled. + +# How to Improve +Ensure tools are enabled in agent mode for file search, terminal access, and web search capabilities. + +# Examples +{{extra.agentName}}: "{{message}}..." + +# Detection Logic +```detect +scan: requests +match: (agentMode == "agent" OR agentName != "") AND toolsUsed.length == 0 +aggregate: count +check: count > thresholds.minSample +examples: "{{messageText | truncate:60}}" +``` diff --git a/packages/ai-coach-analytics/src/rules/auto-approve-terminal.md b/packages/ai-coach-analytics/src/rules/auto-approve-terminal.md new file mode 100644 index 0000000..41da945 --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/auto-approve-terminal.md @@ -0,0 +1,36 @@ +--- +id: auto-approve-terminal +name: Auto-Approved Terminal Commands +group: code-review +severity: medium +scope: requests +requiresIdeContext: true +version: 1 +tags: [review, security, terminal, auto-approve] +thresholds: + minAutoApprove: 20 + minTerminalAutoApprove: 10 +--- + +# Description +Detects terminal commands that were auto-approved without review, which can be risky for destructive operations. + +# When Triggered +{{count}} terminal commands were auto-approved. Blindly running AI-generated commands can be risky. + +# How to Improve +Review terminal commands before execution, especially destructive ones (rm, git push --force, DROP TABLE). Use session-scoped approval cautiously. + +# Examples +$ {{extra.command}} + +# Detection Logic +```detect +scan: requests +match: toolConfirmations.length > 0 +aggregate: count +stats: autoApproveStats(matched) +emitCount: stats.terminalAutoApproved +emitTotal: stats.withConfirmations +check: stats.terminalAutoApproved > thresholds.minTerminalAutoApprove AND stats.autoApprovedTotal > thresholds.minAutoApprove +``` diff --git a/packages/ai-coach-analytics/src/rules/auto-avoidance.md b/packages/ai-coach-analytics/src/rules/auto-avoidance.md new file mode 100644 index 0000000..7d01f8f --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/auto-avoidance.md @@ -0,0 +1,38 @@ +--- +id: auto-avoidance +name: Auto Model Avoidance +group: tool-mastery +severity: medium +scope: requests +version: 1 +tags: [model, auto, routing, cost] +thresholds: + minTopShare: 0.6 + minSample: 30 +--- + +# Description +Detects users who pin a single premium model for every request and never let auto-routing pick a cheaper model when one would suffice. Auto routing handles many requests with smaller models for the same outcome. + +# When Triggered +{{pct}} of requests use {{extra.topModel}} (a premium model) and no requests use auto routing. Pinning a top-tier model for every task overpays on simple work. + +# How to Improve +Switch the default to "auto" or use the model picker. Reserve specific top-tier models (Claude Opus, GPT-5) for hard reasoning, planning, or large-context tasks. For routine edits and questions, the lighter model auto picks is usually enough. + +# Examples +{{extra.topModel}}: {{extra.topCount}} requests ({{pct}}) + +# Detection Logic +```detect +scan: requests +match: modelId != "" +aggregate: count +models: modelStats(matched) +topModel: models.topModel +topCount: models.topCount +hasAutoUsage: countWhere(matched, "modelId", "matches", "(?i)auto") +emitCount: models.topCount +emitTotal: models.total +check: models.topShare > thresholds.minTopShare AND modelTier(models.topModel) >= 1 AND hasAutoUsage == 0 AND models.total > thresholds.minSample +``` diff --git a/packages/ai-coach-analytics/src/rules/broken-flow-state.md b/packages/ai-coach-analytics/src/rules/broken-flow-state.md new file mode 100644 index 0000000..625f703 --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/broken-flow-state.md @@ -0,0 +1,48 @@ +--- +id: broken-flow-state +name: Broken Flow State +group: session-hygiene +severity: medium +scope: sessions +version: 1 +tags: [session, flow, focus, interruptions] +thresholds: + rapidFollowupSec: 30 + sessionMinReqs: 3 + shallowScore: 25 + lowScoreRate: 0.6 + minDays: 5 +--- + +# Description +Detects fragmented coding flow with long pauses between prompts, indicating frequent context switches, interruptions, or multitasking. + +# When Triggered +{{count}}/{{extra.totalDays}} days ({{pct}}) show fragmented flow -- long pauses between prompts, short scattered work blocks. +Average flow score: {{extra.avgScore}}/100. +This anti-pattern indicates frequent context switches, interruptions, or multitasking that prevent deep coding flow. + +# How to Improve +Block 2+ hour uninterrupted time slots for AI-assisted coding. +Close Slack / email / notifications during coding sessions. +Pre-plan your next prompt while the agent works on the current one. +Batch meetings to protect contiguous coding blocks. +Use the Flow State page to find your most productive hours. + +# Examples +{{extra.date}}: avg flow score {{extra.flowScore}}/100 + +# Detection Logic +```detect +scan: sessions +match: requestCount >= thresholds.sessionMinReqs +aggregate: count +flow: flowScoreStats(allSessions, thresholds.sessionMinReqs, thresholds.rapidFollowupSec * 1000) +emitCount: flow.fragmentedDays +emitTotal: flow.totalDays +totalDays: flow.totalDays +avgScore: flow.avgScore +check: flow.lowScoreRate > thresholds.lowScoreRate AND flow.totalDays >= thresholds.minDays +severity: flow.lowScoreRate > 0.8 +examples: {{workspaceName}}: flow disruption detected +``` diff --git a/packages/ai-coach-analytics/src/rules/cache-hit-starvation.md b/packages/ai-coach-analytics/src/rules/cache-hit-starvation.md new file mode 100644 index 0000000..d97ed19 --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/cache-hit-starvation.md @@ -0,0 +1,40 @@ +--- +id: cache-hit-starvation +name: Prompt Cache Starvation +group: tool-mastery +severity: medium +scope: requests +version: 1 +tags: [tokens, cache, context, cost] +thresholds: + minPromptTokens: 5000 + minSample: 20 + minCacheRate: 0.1 +--- + +# Description +Detects requests with large prompts (>5K input tokens) where almost none of the prompt is being served from the model's prompt cache. Low cache hit rates mean every request pays full price for the same prefixes — usually caused by churning instructions, frequent compaction, or unstable system prompts. + +# When Triggered +{{count}} requests had prompts >{{extra.minPromptTokens}} tokens but only {{extra.cachePctLabel}} of input tokens came from cache. Each long-prompt request is paying full price for context the model has seen before. + +# How to Improve +Stabilize the front of your prompts: keep custom instructions short and stable, avoid frequent compaction, prefer file references over pasted code, and avoid clearing chat mid-task. Within a session, repeated prefixes get cached and re-used for free. + +# Examples +{{promptTokens}} prompt tokens · {{coalesce(cacheReadTokens, 0)}} cached + +# Detection Logic +```detect +scan: requests +match: promptTokens > thresholds.minPromptTokens +aggregate: count +totalPrompt: sumField(matched, "promptTokens") +totalCache: sumField(matched, "cacheReadTokens") +cacheRate: totalCache / totalPrompt +cachePctLabel: cacheRate | pct:1 +emitCount: count +emitTotal: count +check: count > thresholds.minSample AND cacheRate < thresholds.minCacheRate +examples: {{promptTokens}} prompt tokens, {{coalesce(cacheReadTokens, 0)}} cached +``` diff --git a/packages/ai-coach-analytics/src/rules/caps-lock.md b/packages/ai-coach-analytics/src/rules/caps-lock.md new file mode 100644 index 0000000..76e1d64 --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/caps-lock.md @@ -0,0 +1,34 @@ +--- +id: caps-lock +name: Caps Lock Rage +group: prompt-quality +severity: medium +scope: requests +version: 1 +tags: [prompt, frustration, caps] +thresholds: + minLength: 10 + capsRate: 0.9 + minReqs: 1 +--- + +# Description +Detects requests written mostly or entirely in CAPS LOCK, indicating high frustration levels. + +# When Triggered +{{count}} requests are written mostly or entirely in CAPS LOCK, indicating high frustration. + +# How to Improve +All-caps messages signal frustration. Step away, take a breath, then return with a calm, structured prompt. Clear communication gets better AI responses. + +# Examples +"{{message}}" + +# Detection Logic +```detect +scan: requests +match: messageLength >= thresholds.minLength AND capsLetterRatio(messageText) >= thresholds.capsRate +aggregate: count +check: count >= thresholds.minReqs +examples: "{{messageText | truncate:80}}" +``` diff --git a/packages/ai-coach-analytics/src/rules/context-engineering-gaps.md b/packages/ai-coach-analytics/src/rules/context-engineering-gaps.md new file mode 100644 index 0000000..e1230d4 --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/context-engineering-gaps.md @@ -0,0 +1,50 @@ +--- +id: context-engineering-gaps +name: Context Engineering Gaps +group: prompt-quality +severity: medium +scope: both +version: 1 +tags: [context, agents, skills, mcp, instructions] +thresholds: + minReqs: 30 + fileRefMinRate: 0.1 + instructionMinRate: 0.05 +--- + +# Description +Audits your context engineering setup: custom agents, skills, MCP tools, file references, and custom instructions. Missing components limit AI effectiveness. + +# When Triggered +{{count}} of 5 context engineering signals missing. Your AI lacks the context to be maximally effective. + +# How to Improve +Level up your context engineering: create AGENTS.md for custom agents, SKILL.md for domain knowledge, connect MCP tools, use #file references, and add .instructions.md with project conventions. + +# Examples +{{extra.gapCount}} of 5 context engineering signals missing + +# Detection Logic +```detect +scan: requests +match: true +aggregate: count +reqCount: count +hasSubAgents: someWhere(allReqs, "agentName", "!=", "") AND \ + someWhere(allReqs, "agentName", "!=", "copilot") AND \ + someWhere(allReqs, "agentMode", "agent") +hasSkills: flatCount(allReqs, "skillsUsed") > 0 +hasMcp: flatSomeWhere(allReqs, "toolsUsed", ".", "mcp_", "startsWith") +fileRefRate: countWhere(allReqs, "referencedFiles.length", ">", 0) / reqCount +instrRate: countWhere(allReqs, "customInstructions.length", ">", 0) / reqCount +gap1: hasSubAgents == 0 +gap2: hasSkills == 0 +gap3: hasMcp == 0 +gap4: fileRefRate < thresholds.fileRefMinRate +gap5: instrRate < thresholds.instructionMinRate +gapCount: gap1 + gap2 + gap3 + gap4 + gap5 +emitCount: gapCount +emitTotal: 5 +check: gapCount > 0 AND reqCount >= thresholds.minReqs +severity: gapCount >= 4 +``` diff --git a/packages/ai-coach-analytics/src/rules/copy-paste-blindness.md b/packages/ai-coach-analytics/src/rules/copy-paste-blindness.md new file mode 100644 index 0000000..a91c634 --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/copy-paste-blindness.md @@ -0,0 +1,35 @@ +--- +id: copy-paste-blindness +name: Copy-Paste Blindness +group: code-review +severity: high +scope: sessions +version: 1 +tags: [review, copy-paste, blindness] +thresholds: + minAiLoc: 50 + minSessions: 3 +--- + +# Description +Detects sessions with large AI-generated code blocks that have no follow-up refinement, suggesting code is accepted without review. + +# When Triggered +{{count}} sessions have large AI-generated code blocks with no follow-up refinement. Code may be accepted without review. + +# How to Improve +Always review AI-generated code before accepting. Ask follow-up questions to refine, test, and understand the output. + +# Examples +{{extra.workspace}}: {{extra.aiLoc}} AI LOC, {{extra.messageCount}} messages, no refinement + +# Detection Logic +```detect +scan: sessions +match: requestCount >= 2 AND flatSumField(requests, "aiCode", "loc") >= thresholds.minAiLoc AND NOT (\ + anyWhere(slice(requests, 1), "messageText", "matches", "(?i)\\b(change|fix|modify|update|refactor|wrong|instead|actually|revert|redo|try again)\\b") OR \ + someWhere(slice(requests, 1), "editedFiles.length", ">", 0)) +aggregate: count +check: count >= thresholds.minSessions +examples: {{workspaceName}}: {{flatSumField(requests, "aiCode", "loc")}} AI LoC, no refinement +``` diff --git a/packages/ai-coach-analytics/src/rules/excessive-file-context.md b/packages/ai-coach-analytics/src/rules/excessive-file-context.md new file mode 100644 index 0000000..80bf6e8 --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/excessive-file-context.md @@ -0,0 +1,39 @@ +--- +id: excessive-file-context +name: Excessive File Context +group: prompt-quality +severity: medium +scope: requests +version: 1 +tags: [context, files, tokens, cost] +thresholds: + minFiles: 30 + minOutliers: 10 + maxRatio: 0.005 +--- + +# Description +Detects outlier requests that attach a very large number of files to the prompt. Each referenced file expands the input context — and the agent rarely reads all of them. Bulk-attaching files is a common pattern when the user isn't sure which files are relevant; it works, but pays for context that's mostly never used. + +# When Triggered +{{count}} request(s) attached ≥{{thresholds.minFiles}} files to the prompt (largest: {{extra.maxFiles}} files). The model only reads a fraction of large attachments — the rest is paid-for context that contributes nothing. + +# How to Improve +Be selective: attach only the 3–5 files most relevant to the task. Use `#codebase` or `#file:` patterns to let the model search on demand. For exploration tasks, ask the agent to `grep` first and read results into context only as needed. + +# Examples +{{length(referencedFiles)}} files: "{{messageText | truncate:60}}" + +# Detection Logic +```detect +scan: requests +match: length(referencedFiles) >= thresholds.minFiles +aggregate: ratio +stats: excessFileContextStats(allReqs, thresholds.minFiles) +maxFiles: stats.maxFiles +p95Files: stats.p95Files +emitCount: stats.outlierCount +emitTotal: stats.totalReqs +check: stats.outlierCount >= thresholds.minOutliers AND stats.ratio >= thresholds.maxRatio +examples: {{length(referencedFiles)}} files: "{{messageText | truncate:60}}" +``` diff --git a/packages/ai-coach-analytics/src/rules/frustration-signals.md b/packages/ai-coach-analytics/src/rules/frustration-signals.md new file mode 100644 index 0000000..8b6950e --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/frustration-signals.md @@ -0,0 +1,44 @@ +--- +id: frustration-signals +name: Frustration Signals +group: prompt-quality +severity: medium +scope: requests +version: 1 +tags: [prompt, frustration, caps, punctuation] +thresholds: + capsRate: 0.4 + minWords: 3 + minReqs: 2 +patterns: + frustration: ["!{3,}", "\\?{3,}", "wtf|come on|why won't", "this is broken|doesn't work"] +--- + +# Description +Detects requests showing frustration indicators like excessive punctuation (!!!?, ???) or ALL CAPS writing. + +# When Triggered +{{count}} requests show frustration indicators (excessive punctuation, ALL CAPS). This usually means the approach isn't working. + +# How to Improve +When frustrated, step back and change strategy. Start a new session, rephrase the problem, or break it into smaller pieces instead of escalating the same prompt. + +# Examples +"{{message}}" + +# Detection Logic +```detect +scan: requests +match: messageLength >= 10 AND (matchesAny(messageText, patterns.frustration) OR capsWordRatio(messageText, thresholds.minWords) >= thresholds.capsRate) +aggregate: count +check: count >= thresholds.minReqs +examples: "{{messageText | truncate:80}}" +``` + +# Tests +```test +{messageText: "WHY WONT THIS WORK???!!!", messageLength: 25} -> triggered +{messageText: "Please refactor the auth module", messageLength: 31} -> clean +{messageText: "THIS IS SO BROKEN FIX IT NOW", messageLength: 28} -> triggered +{messageText: "Add error handling to the API endpoint", messageLength: 38} -> clean +``` diff --git a/packages/ai-coach-analytics/src/rules/high-cancellation.md b/packages/ai-coach-analytics/src/rules/high-cancellation.md new file mode 100644 index 0000000..7db494c --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/high-cancellation.md @@ -0,0 +1,33 @@ +--- +id: high-cancellation +name: Excessive Cancellations +group: session-hygiene +severity: medium +scope: requests +version: 1 +tags: [session, cancellation, waste] +thresholds: + maxCancelRate: 0.15 + highSeverityRate: 0.3 +--- + +# Description +Detects a high rate of cancelled requests, which wastes premium quota and indicates unclear prompting. + +# When Triggered +{{count}} of {{total}} requests cancelled ({{pct}}). This wastes premium quota. + +# How to Improve +Write clearer, more specific prompts. Wait for responses instead of cancelling prematurely. + +# Examples +"{{message}}..." + +# Detection Logic +```detect +scan: requests +match: isCanceled == true +aggregate: ratio +check: ratio > thresholds.maxCancelRate +examples: "{{messageText | clip:80}}" +``` diff --git a/packages/ai-coach-analytics/src/rules/instruction-bloat.md b/packages/ai-coach-analytics/src/rules/instruction-bloat.md new file mode 100644 index 0000000..f51a6d0 --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/instruction-bloat.md @@ -0,0 +1,41 @@ +--- +id: instruction-bloat +name: Instruction Bloat +group: prompt-quality +severity: medium +scope: sessions +requiresIdeContext: true +version: 1 +tags: [tokens, instructions, context, cost] +thresholds: + maxBytes: 4000 + minBloated: 1 +--- + +# Description +Detects oversized `.github/copilot-instructions.md` (or equivalent) files. Custom instructions are prepended to **every** request's system prompt — large files inflate input tokens on every turn, often by thousands of tokens that are never relevant to the current task. + +# When Triggered +{{extra.bloatedSessions}} workspace(s) have custom-instruction files larger than {{thresholds.maxBytes}} bytes (largest: {{extra.maxBytes}} bytes). Every request in those workspaces pays the bloat cost on input tokens. + +# How to Improve +Trim `.github/copilot-instructions.md` to the essentials: language/framework conventions, code style, "do not" rules, and pointers to longer docs. Move long examples and rationale into separate files referenced via `#file:`. Keep the always-on payload under ~4 KB. + +# Examples +{{extra.maxBytes}} bytes — largest custom-instructions file across {{extra.totalSessions}} workspace(s) + +# Detection Logic +```detect +scan: sessions +match: true +aggregate: count +stats: instructionBloatStats(allSessions, thresholds.maxBytes) +bloatedSessions: stats.bloatedSessions +maxBytes: stats.maxBytes +totalSessions: stats.totalSessions +withInstructionsCount: stats.withInstructionsCount +emitCount: stats.bloatedSessions +emitTotal: stats.totalSessions +check: stats.bloatedSessions >= thresholds.minBloated +examples: {{maxBytes}} bytes (workspace-level) +``` diff --git a/packages/ai-coach-analytics/src/rules/late-night-coding.md b/packages/ai-coach-analytics/src/rules/late-night-coding.md new file mode 100644 index 0000000..de367b2 --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/late-night-coding.md @@ -0,0 +1,33 @@ +--- +id: late-night-coding +name: Late-Night Coding +group: session-hygiene +severity: low +scope: requests +version: 1 +tags: [session, health, hours] +thresholds: + lateNightHour: 5 + minSample: 10 +--- + +# Description +Detects requests made between midnight and early morning. Late-night coding correlates with more bugs and lower quality output. + +# When Triggered +{{count}} requests were made between midnight and {{extra.lateNightHour}}am. Late-night coding correlates with more bugs and lower quality. + +# How to Improve +Consider establishing healthier work hours. Quality drops significantly when fatigued. + +# Examples +{{extra.timestamp}}: "{{message}}..." + +# Detection Logic +```detect +scan: requests +match: timestamp > 0 AND hour(timestamp) >= 0 AND hour(timestamp) < thresholds.lateNightHour +aggregate: count +check: count > thresholds.minSample +examples: "{{messageText | truncate:50}}" +``` diff --git a/packages/ai-coach-analytics/src/rules/lazy-prompting.md b/packages/ai-coach-analytics/src/rules/lazy-prompting.md new file mode 100644 index 0000000..21a7acd --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/lazy-prompting.md @@ -0,0 +1,41 @@ +--- +id: lazy-prompting +name: Lazy Prompting +group: prompt-quality +severity: medium +scope: requests +version: 1 +tags: [prompt, quality, short] +thresholds: + minChars: 30 + maxRatio: 0.3 + minSample: 10 +--- + +# Description +Detects requests with very short prompts that lack sufficient context for the AI to produce quality results. + +# When Triggered +{{count}} requests ({{pct}}) are under {{extra.minChars}} characters. Very short prompts often produce poor results. + +# How to Improve +Provide more context in your prompts: describe the intent, constraints, and expected output format. + +# Examples +"{{message}}" ({{extra.charCount}} chars) + +# Detection Logic +```detect +scan: requests +match: messageLength < thresholds.minChars AND messageLength > 0 +aggregate: ratio +check: ratio > thresholds.maxRatio AND count > thresholds.minSample +examples: "{{messageText | truncate:80}}" ({{messageLength}} chars) +``` + +# Tests +```test +{messageText: "fix bug", messageLength: 7} -> triggered +{messageText: "Refactor the authentication middleware to use JWT tokens and add refresh token rotation", messageLength: 88} -> clean +{messageText: "", messageLength: 0} -> clean +``` diff --git a/packages/ai-coach-analytics/src/rules/low-constraint-usage.md b/packages/ai-coach-analytics/src/rules/low-constraint-usage.md new file mode 100644 index 0000000..d962159 --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/low-constraint-usage.md @@ -0,0 +1,41 @@ +--- +id: low-constraint-usage +name: Low Constraint Usage +group: prompt-quality +severity: medium +scope: requests +version: 1 +tags: [prompt, constraints, quality] +thresholds: + minReqs: 30 + minMessageLength: 40 + constraintRate: 0.08 +patterns: + constraintKeywords: ["\\b(do not|don't|must not|never|without|avoid|only|strictly|limit to|at most|at least|no more than|require|restrict|exclude|ensure|must|shall)\\b"] +--- + +# Description +Detects prompts that lack constraint keywords (do not, must, avoid, only, etc.). Constraints narrow AI output, reduce hallucinations, and produce higher-quality code. + +# When Triggered +Only {{extra.constraintPct}}% of prompts include constraints ({{extra.withConstraints}}/{{extra.substantialReqs}}). Constraints narrow AI output, reduce hallucinations, and produce higher-quality code. + +# How to Improve +Add explicit constraints to prompts: "do not use class components", "only use async/await", "limit to 50 lines", "avoid external dependencies". Negative constraints force the model out of boilerplate patterns and into more precise solutions. + +# Examples +"{{message}}..." (no constraints) + +# Detection Logic +```detect +scan: requests +match: messageLength >= thresholds.minMessageLength AND NOT \ + matches(messageText, "(?i)\\b(do not|don't|must not|never|without|avoid|only|strictly|limit to|at most|at least|no more than|require|restrict|exclude|ensure|must|shall|should not)\\b") +aggregate: count +substantialTotal: countWhere(all, "messageLength", ">=", thresholds.minMessageLength) +withConstraints: substantialTotal - count +substantialReqs: substantialTotal +constraintPct: round((substantialTotal - count) / substantialTotal * 100) +check: substantialTotal >= thresholds.minReqs AND count / substantialTotal > (1 - thresholds.constraintRate) +examples: "{{messageText | truncate:80}}" (no constraints) +``` diff --git a/packages/ai-coach-analytics/src/rules/low-markdown-ratio.md b/packages/ai-coach-analytics/src/rules/low-markdown-ratio.md new file mode 100644 index 0000000..ca1c8c6 --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/low-markdown-ratio.md @@ -0,0 +1,45 @@ +--- +id: low-markdown-ratio +name: Low Markdown Output Ratio +group: prompt-quality +severity: medium +scope: sessions +version: 1 +tags: [prompt, markdown, documentation] +thresholds: + markdownRatio: 0.05 + minTotalLoc: 100 + minWorkspaces: 1 +fileTypes: + documentation: [markdown, md, mdx, txt, rst, adoc] +patterns: + docFileExtensions: [md, mdx, txt, rst, adoc] +--- + +# Description +Detects workspaces that produce almost no markdown output, suggesting specs, plans, and documentation are skipped before coding. + +# When Triggered +{{count}} workspace(s) produce almost no markdown ({{extra.overallPct}}% of AI output). {{extra.totalCodeLoc}} LoC of code vs {{extra.totalMdLoc}} LoC of markdown. This suggests you skip writing specs, plans, and documentation before coding. + +# How to Improve +Adopt spec-driven development: ask Copilot to draft a spec, plan, or design doc before writing code. Even a short markdown outline dramatically improves code quality and reduces iteration cycles. Try starting sessions with "Write a brief spec for..." or "Draft an implementation plan for...". + +# Examples +{{extra.workspace}}: {{extra.codeLoc}} code LoC, {{extra.mdLoc}} markdown LoC ({{extra.ratioPct}}%) + +# Detection Logic +```detect +scan: sessions +match: true +aggregate: count +md: mdRatioByWorkspace(allSessions, thresholds.minTotalLoc, ["markdown", "md"]) +emitCount: md.lowCount +emitTotal: md.totalWorkspaces +overallPct: round(md.overallRatio * 100) +totalCodeLoc: md.totalCodeLoc +totalMdLoc: md.totalMdLoc +workspaces: md.workspaces +check: md.lowCount >= thresholds.minWorkspaces +severity: md.overallRatio < 0.02 +``` diff --git a/packages/ai-coach-analytics/src/rules/mcp-tool-bloat.md b/packages/ai-coach-analytics/src/rules/mcp-tool-bloat.md new file mode 100644 index 0000000..3249406 --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/mcp-tool-bloat.md @@ -0,0 +1,33 @@ +--- +id: mcp-tool-bloat +name: Tool / MCP Bloat +group: tool-mastery +severity: medium +scope: sessions +version: 2 +tags: [tools, mcp, context-window] +thresholds: + maxToolsPerSession: 40 + minSessions: 3 +--- + +# Description +Detects sessions with an unusually large number of distinct tools invoked — a proxy for oversized tool catalogs that inflate every request's system prompt. Each registered tool adds tokens regardless of whether it's used. + +# When Triggered +{{count}} session(s) used more than {{thresholds.maxToolsPerSession}} distinct tools. Large tool sets add silent overhead to every prompt. + +# How to Improve +Trim the active toolset: disable rarely-used MCP servers, scope tool sets per workspace, and use tool groups so only relevant tools are loaded for the task at hand. Aim for under 40 active tools per session. + +# Examples +{{flatUnique(reqs, "toolsUsed")}} tools in one session + +# Detection Logic +```detect +scan: sessions +match: flatUnique(reqs, "toolsUsed") > thresholds.maxToolsPerSession +aggregate: count +check: count >= thresholds.minSessions +examples: {{flatUnique(reqs, "toolsUsed")}} tools +``` diff --git a/packages/ai-coach-analytics/src/rules/mega-sessions.md b/packages/ai-coach-analytics/src/rules/mega-sessions.md new file mode 100644 index 0000000..09e57c7 --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/mega-sessions.md @@ -0,0 +1,32 @@ +--- +id: mega-sessions +name: Mega Sessions +group: session-hygiene +severity: high +scope: sessions +version: 1 +tags: [session, length, context] +thresholds: + maxMessages: 50 +--- + +# Description +Detects sessions with an excessive number of messages. Long sessions degrade context quality and response accuracy. + +# When Triggered +{{count}} session(s) have {{extra.maxMessages}}+ messages. Long sessions degrade context quality and response accuracy. + +# How to Improve +Start new sessions periodically. Break large tasks into focused conversations of 15-25 messages. + +# Examples +{{extra.workspace}}: {{extra.messageCount}} messages + +# Detection Logic +```detect +scan: sessions +match: requestCount >= thresholds.maxMessages +aggregate: count +check: count > 0 +examples: {{workspaceName}}: {{requestCount}} messages +``` diff --git a/packages/ai-coach-analytics/src/rules/model-overreliance.md b/packages/ai-coach-analytics/src/rules/model-overreliance.md new file mode 100644 index 0000000..6ef06ab --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/model-overreliance.md @@ -0,0 +1,37 @@ +--- +id: model-overreliance +name: Model Overreliance +group: tool-mastery +severity: medium +scope: requests +version: 1 +tags: [tools, model, diversity] +thresholds: + maxTopModelRate: 0.8 + minSample: 10 + minModels: 3 +--- + +# Description +Detects when the vast majority of requests use a single model, missing opportunities to use lighter models for simple tasks. + +# When Triggered +{{pct}} of requests use {{extra.topModel}}. Different tasks benefit from different models. + +# How to Improve +Use lighter models (gpt-4.1-mini, gemini-flash) for simple tasks to save premium quota and get faster responses. + +# Examples +{{extra.model}}: {{extra.reqCount}} requests + +# Detection Logic +```detect +scan: requests +match: true +aggregate: count +models: modelStats(allReqs) +emitCount: models.topCount +emitTotal: models.total +topModel: models.topModel +check: models.topShare > thresholds.maxTopModelRate AND models.modelCount < thresholds.minModels AND models.total > thresholds.minSample +``` diff --git a/packages/ai-coach-analytics/src/rules/no-custom-instructions.md b/packages/ai-coach-analytics/src/rules/no-custom-instructions.md new file mode 100644 index 0000000..422f3b8 --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/no-custom-instructions.md @@ -0,0 +1,36 @@ +--- +id: no-custom-instructions +name: No Custom Instructions +group: tool-mastery +severity: medium +scope: requests +requiresIdeContext: true +version: 1 +tags: [tools, instructions, personalization] +thresholds: + minRate: 0.05 + minReqs: 20 +--- + +# Description +Detects when very few requests use custom instructions, missing out on personalized and project-specific responses. + +# When Triggered +Only {{extra.usagePct}}% of requests use custom instructions ({{extra.withInstructions}}/{{total}}). Missing out on personalized responses. + +# How to Improve +Create a .github/copilot-instructions.md or .instructions.md file in your workspace to give Copilot persistent context about your project conventions, stack, and coding style. + +# Examples +{{extra.withInstructions}} of {{total}} requests had custom instructions + +# Detection Logic +```detect +scan: requests +match: customInstructions.length == 0 +aggregate: count +usageRate: (total - count) / total +withInstructions: total - count +usagePct: round(usageRate * 100) +check: usageRate < thresholds.minRate AND total > thresholds.minReqs +``` diff --git a/packages/ai-coach-analytics/src/rules/no-devcontainer.md b/packages/ai-coach-analytics/src/rules/no-devcontainer.md new file mode 100644 index 0000000..7af49eb --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/no-devcontainer.md @@ -0,0 +1,36 @@ +--- +id: no-devcontainer +name: Unsandboxed Terminal Execution +group: code-review +severity: medium +scope: both +requiresIdeContext: true +version: 1 +tags: [review, security, devcontainer, terminal] +thresholds: + minTerminalReqs: 20 + terminalRate: 0.2 +--- + +# Description +Detects terminal commands that ran on the host machine instead of inside a devcontainer. Per-session detection: a session is considered sandboxed when its runtime data shows `/workspaces/...` paths (the Codespaces / Remote-Containers convention). Sessions without that evidence are treated as host-execution and counted here. + +# When Triggered +{{count}} terminal commands ({{pct}} of VS Code requests) ran directly on your host machine without a devcontainer. Agent-driven terminal commands can modify system state, install packages, or delete files outside the project. + +# How to Improve +Set up a .devcontainer/devcontainer.json (or use a Codespace) to sandbox AI-driven terminal execution. Devcontainers isolate builds, installs, and destructive commands from your host OS — especially important when using agent mode with auto-approved terminal access. + +# Examples +$ {{extra.command}} + +# Detection Logic +```detect +scan: sessions +match: true +aggregate: count +dc: devcontainerStats(allSessions, allReqs) +emitCount: dc.terminalReqs +emitTotal: dc.vscodeReqs +check: dc.terminalReqs >= thresholds.minTerminalReqs AND dc.terminalRate >= thresholds.terminalRate +``` diff --git a/packages/ai-coach-analytics/src/rules/no-file-context.md b/packages/ai-coach-analytics/src/rules/no-file-context.md new file mode 100644 index 0000000..1590daa --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/no-file-context.md @@ -0,0 +1,34 @@ +--- +id: no-file-context +name: Missing File Context +group: prompt-quality +severity: medium +scope: requests +requiresIdeContext: true +version: 1 +tags: [prompt, context, files] +thresholds: + maxNoContextRate: 0.7 + minSample: 10 +--- + +# Description +Detects requests that have no file references, meaning Copilot cannot see the relevant code context. + +# When Triggered +{{pct}} of requests have no file references. Copilot gives better answers with file context. + +# How to Improve +Use #file to reference relevant files, or open files in the editor so Copilot can use them as context. + +# Examples +"{{message}}..." + +# Detection Logic +```detect +scan: requests +match: referencedFiles.length == 0 AND editedFiles.length == 0 +aggregate: ratio +check: ratio > thresholds.maxNoContextRate AND count > thresholds.minSample +examples: "{{messageText | clip:80}}" +``` diff --git a/packages/ai-coach-analytics/src/rules/no-language-exploration.md b/packages/ai-coach-analytics/src/rules/no-language-exploration.md new file mode 100644 index 0000000..f01d956 --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/no-language-exploration.md @@ -0,0 +1,38 @@ +--- +id: no-language-exploration +name: No Language Exploration +group: code-review +severity: low +scope: requests +version: 1 +tags: [review, languages, exploration] +thresholds: + minSample: 10 + minWeeks: 4 + recentWeeks: 8 +--- + +# Description +Detects when no new programming languages have been explored recently, despite AI being a learning accelerator. + +# When Triggered +You've used {{extra.totalLanguages}} languages total, but haven't explored a new one in {{count}} weeks. AI is a learning accelerator -- use it to try new languages and frameworks. + +# How to Improve +AI coding assistants dramatically lower the barrier to learning new languages. Try asking Copilot to help you build something small in a language you haven't used before. Start with "Write a simple HTTP server in [Go/Rust/Elixir]" to break the ice. + +# Examples +{{extra.languageList}} + +# Detection Logic +```detect +scan: requests +match: timestamp > 0 +aggregate: count +lang: langExplorationWeeks(allReqs) +emitCount: lang.weeksSinceNew +emitTotal: lang.totalWeeks +totalLanguages: lang.totalLangs +check: lang.recentNew == 0 AND lang.totalWeeks >= thresholds.minWeeks +severity: lang.weeksSinceNew > 12 +``` diff --git a/packages/ai-coach-analytics/src/rules/no-plan-mode.md b/packages/ai-coach-analytics/src/rules/no-plan-mode.md new file mode 100644 index 0000000..bc3e19d --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/no-plan-mode.md @@ -0,0 +1,40 @@ +--- +id: no-plan-mode +name: Never Uses Plan Mode +group: tool-mastery +severity: medium +scope: requests +requiresIdeContext: true +version: 1 +tags: [tools, planning, agent] +thresholds: + minReqs: 30 + agentRate: 0.3 +--- + +# Description +Detects heavy agentic usage with no use of plan mode, which helps the agent understand scope before implementation. + +# When Triggered +{{extra.agenticReqs}} agentic requests but no use of plan mode. Jumping straight to implementation often leads to wrong approaches. + +# How to Improve +Use plan mode (or /plan) before complex tasks. Planning helps the agent understand scope, break down work, and avoid wasted iterations. + +# Examples +Switch to Plan mode in the mode picker before starting large features +Use /plan to outline an approach before coding +Plan first, then switch to Agent mode to execute + +# Detection Logic +```detect +scan: requests +match: agentMode == "agent" OR agentName != "" +aggregate: count +agentRatio: count / total +planUsage: someWhere(all, "slashCommand", "plan") OR \ + someWhere(all, "agentMode", "matches", "(?i)plan"slashCommand", "plan") OR \ + someWhere(all, "agentMode", "matches", "(?i)plan") +agenticReqs: count +check: planUsage == 0 AND total >= thresholds.minReqs AND agentRatio >= thresholds.agentRate +``` diff --git a/packages/ai-coach-analytics/src/rules/no-skills.md b/packages/ai-coach-analytics/src/rules/no-skills.md new file mode 100644 index 0000000..b569245 --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/no-skills.md @@ -0,0 +1,33 @@ +--- +id: no-skills +name: No Skills Usage +group: tool-mastery +severity: low +scope: requests +requiresIdeContext: true +version: 1 +tags: [tools, skills, domain] +thresholds: + minReqs: 50 +--- + +# Description +Detects when no requests use Copilot skills, missing out on specialized domain knowledge. + +# When Triggered +No requests use Copilot skills. Skills provide specialized domain knowledge beyond general coding. + +# How to Improve +Explore available skills in your IDE. Skills can help with specific frameworks, cloud providers, and development workflows. + +# Examples +Skills extend Copilot with domain expertise +Check VS Code extensions for available skills + +# Detection Logic +```detect +scan: requests +match: skillsUsed.length == 0 +aggregate: count +check: count == total AND total > thresholds.minReqs +``` diff --git a/packages/ai-coach-analytics/src/rules/no-slash-commands.md b/packages/ai-coach-analytics/src/rules/no-slash-commands.md new file mode 100644 index 0000000..5f0985f --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/no-slash-commands.md @@ -0,0 +1,37 @@ +--- +id: no-slash-commands +name: No Slash Commands +group: tool-mastery +severity: low +scope: requests +requiresIdeContext: true +version: 1 +tags: [tools, slash, commands] +thresholds: + minRate: 0.02 + minReqs: 20 +--- + +# Description +Detects low usage of slash commands, which produce more targeted responses than freeform prompts. + +# When Triggered +Only {{extra.withSlash}} of {{total}} requests use slash commands. Slash commands produce more targeted responses. + +# How to Improve +Try /fix for bugs, /explain for understanding code, /tests for test generation, /doc for documentation. + +# Examples +/fix - Fix bugs in selected code +/explain - Explain how code works +/tests - Generate unit tests + +# Detection Logic +```detect +scan: requests +match: slashCommand == "" +aggregate: count +usageRate: (total - count) / total +withSlash: total - count +check: usageRate < thresholds.minRate AND total > thresholds.minReqs +``` diff --git a/packages/ai-coach-analytics/src/rules/no-spec-driven-development.md b/packages/ai-coach-analytics/src/rules/no-spec-driven-development.md new file mode 100644 index 0000000..4de9a4f --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/no-spec-driven-development.md @@ -0,0 +1,57 @@ +--- +id: no-spec-driven-development +name: No Spec-Driven Development +group: prompt-quality +severity: medium +scope: sessions +version: 1 +tags: [prompt, spec, planning] +thresholds: + minAgentSessions: 5 + specRate: 0.2 + planRate: 0.15 +patterns: + specFileExts: "(?i)\\.(md|txt|spec|prd|design|plan|rfc|adoc)$" + specKeywords: "(?i)\\b(spec|requirements?|acceptance criteria|design doc|PRD|RFC|plan file|constraint|must|should|ensure)\\b" + bulletList: "(?m)^[-*]\\s" + numberedList: "(?m)^\\d+[.)]\\s" + headings: "(?m)^#+\\s" +--- + +# Description +Detects when few sessions start with specs, plans, or structured requirements. Spec-first development consistently beats vibe-coding. + +# When Triggered +Only {{extra.specDrivenCount}} of {{extra.totalSessions}} sessions ({{extra.specPct}}%) start with specs, plans, or structured requirements. Spec-first development consistently beats vibe-coding. + +# How to Improve +Adopt Spec-Driven Development (SDD): write a brief spec before coding. +How sessions are classified: +- Spec-driven: first prompt references a .md/.spec file, contains structured bullet points/numbered lists, uses keywords like "requirements", "acceptance criteria", "must", "ensure", or starts in plan mode. +- Planning: any request uses plan mode (/plan) or contains planning keywords (plan, architect, design, outline, roadmap). +- Unstructured: everything else — typically vague, single-sentence prompts that lead to more iterations. +Start each session with: 1) What you're building, 2) Acceptance criteria, 3) Constraints. Even 3 bullet points dramatically improve AI output quality. + +# Examples +{{extra.workspace}}: "{{message}}..." + +# Detection Logic +```detect +scan: sessions +match: requestCount >= 3 AND NOT (\ + anyMatch(first(requests).referencedFiles, patterns.specFileExts) OR \ + matches(first(requests).messageText, patterns.specKeywords) OR \ + (matches(first(requests).messageText, patterns.bulletList) AND lineCount(first(requests).messageText) >= 3) OR \ + (matches(first(requests).messageText, patterns.numberedList) AND lineCount(first(requests).messageText) >= 3) OR \ + matches(first(requests).messageText, patterns.headings) OR \ + first(requests).slashCommand == "plan" OR \ + contains(str(first(requests).agentMode), "plan")) +aggregate: count +specSessionTotal: countWhere(all, "requestCount", ">=", 3) +specRate: (specSessionTotal - count) / specSessionTotal +specDrivenCount: specSessionTotal - count +totalSessions: specSessionTotal +specPct: round(specRate * 100) +check: specSessionTotal >= thresholds.minAgentSessions AND specRate < thresholds.specRate +examples: {{workspaceName}}: not spec-driven +``` diff --git a/packages/ai-coach-analytics/src/rules/no-spec-structure.md b/packages/ai-coach-analytics/src/rules/no-spec-structure.md new file mode 100644 index 0000000..757cbf6 --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/no-spec-structure.md @@ -0,0 +1,40 @@ +--- +id: no-spec-structure +name: Unstructured Task Starts +group: prompt-quality +severity: medium +scope: sessions +version: 1 +tags: [prompt, structure, agentic] +thresholds: + minAgentSessions: 5 + structuredRate: 0.15 +--- + +# Description +Detects agentic sessions that start with vague, unstructured prompts lacking bullet points, requirements, or acceptance criteria. + +# When Triggered +{{count}} of {{extra.agentSessions}} agentic sessions start with vague, unstructured prompts. No bullet points, requirements, or acceptance criteria. + +# How to Improve +Start agentic sessions with structured specs: use bullet points, numbered requirements, or acceptance criteria. The more specific the first prompt, the better the result. + +# Examples +{{extra.workspace}}: "{{message}}..." + +# Detection Logic +```detect +scan: sessions +match: requestCount >= 3 AND someWhere(requests, "agentMode", "agent") AND NOT (\ + matches(first(requests).messageText, "(?m)^[-*]\\s") OR \ + matches(first(requests).messageText, "(?m)^\\d+[.)]\\s") OR \ + matches(first(requests).messageText, "(?m)^#+\\s") OR \ + matches(first(requests).messageText, "(?i)\\b(requirements?|spec|acceptance criteria|user stories?|given|when|then|should|must)\\b") OR \ + lineCount(first(requests).messageText) >= 4) +aggregate: count +agentSessionTotal: countWhere(all, "requestCount", ">=", 3) +agentSessions: agentSessionTotal +check: agentSessionTotal >= thresholds.minAgentSessions AND count / agentSessionTotal > (1 - thresholds.structuredRate) +examples: {{workspaceName}}: unstructured first prompt +``` diff --git a/packages/ai-coach-analytics/src/rules/premium-for-lookup-questions.md b/packages/ai-coach-analytics/src/rules/premium-for-lookup-questions.md new file mode 100644 index 0000000..87f0f74 --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/premium-for-lookup-questions.md @@ -0,0 +1,34 @@ +--- +id: premium-for-lookup-questions +name: Premium Model for Lookup Questions +group: tool-mastery +severity: medium +scope: requests +version: 1 +tags: [model, lookup, ask, cost] +thresholds: + minSample: 10 + maxRatio: 0.1 + maxMessageLength: 120 +--- + +# Description +Detects "lookup-style" questions ("what is X?", "where is Y?", "how do I Z?", "explain Y") that get routed to a premium model. These factual one-shot questions rarely need top-tier reasoning — `auto` or a base model usually returns the same answer for a fraction of the cost. + +# When Triggered +{{count}} lookup-style questions ({{pct}}) used a premium model. Short factual questions almost never benefit from premium reasoning — they overpay on every request. + +# How to Improve +Switch the default model to `auto` so lightweight questions are routed cheaply. Reserve premium models for tasks that need actual reasoning: planning, debugging, multi-step refactors. Custom-instruct your IDE to "use the cheapest model that can answer this" for short prompts. + +# Examples +{{normalizeModel(modelId)}}: "{{messageText | truncate:60}}" + +# Detection Logic +```detect +scan: requests +match: modelTier(modelId) >= 1 AND messageLength > 0 AND messageLength < thresholds.maxMessageLength AND length(aiCode) == 0 AND length(toolsUsed) == 0 AND matches(messageText, "(?i)^\\s*(what(?:'s| is| are)|where(?:'s| is| are)|how do (?:i|you)|explain|why (?:does|is|are)|when (?:should|do)|which|tell me about|define)\\b") +aggregate: ratio +check: ratio > thresholds.maxRatio AND count > thresholds.minSample +examples: {{normalizeModel(modelId)}}: "{{messageText | truncate:60}}" +``` diff --git a/packages/ai-coach-analytics/src/rules/premium-waste.md b/packages/ai-coach-analytics/src/rules/premium-waste.md new file mode 100644 index 0000000..b412fcc --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/premium-waste.md @@ -0,0 +1,33 @@ +--- +id: premium-waste +name: Premium Model Waste +group: tool-mastery +severity: medium +scope: requests +version: 1 +tags: [tools, model, waste, premium] +thresholds: + minSample: 10 + maxMessageLength: 50 +--- + +# Description +Detects simple requests (short prompt, no code output) that use premium models unnecessarily. + +# When Triggered +{{count}} simple requests (short prompt, no code output) used premium models. + +# How to Improve +Use lighter models for quick questions and simple tasks. Reserve premium models for complex code generation. + +# Examples +{{extra.model}}: "{{message}}..." + +# Detection Logic +```detect +scan: requests +match: modelTier(modelId) >= 1 AND messageLength < thresholds.maxMessageLength AND messageLength > 0 AND aiCode.length == 0 +aggregate: count +check: count > thresholds.minSample +examples: {{normalizeModel(modelId)}}: "{{messageText | truncate:50}}" +``` diff --git a/packages/ai-coach-analytics/src/rules/profanity.md b/packages/ai-coach-analytics/src/rules/profanity.md new file mode 100644 index 0000000..9148688 --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/profanity.md @@ -0,0 +1,32 @@ +--- +id: profanity +name: Hostile Language +group: prompt-quality +severity: medium +scope: requests +version: 1 +tags: [prompt, profanity, hostile] +thresholds: + minReqs: 1 +--- + +# Description +Detects requests containing profanity or hostile language, which usually signals deep frustration with the tool. + +# When Triggered +{{count}} requests contain profanity or hostile language. This usually signals deep frustration with the tool. + +# How to Improve +When you catch yourself using hostile language, take a break. Start a fresh session, rephrase the problem from scratch, or switch to a different approach entirely. + +# Examples +{{extra.highlighted}} in: "{{message}}" + +# Detection Logic +```detect +scan: requests +match: messageLength > 0 AND hasProfanity(messageText) +aggregate: count +check: count >= thresholds.minReqs +examples: "{{messageText | truncate:60}}" +``` diff --git a/packages/ai-coach-analytics/src/rules/reasoning-effort-overuse.md b/packages/ai-coach-analytics/src/rules/reasoning-effort-overuse.md new file mode 100644 index 0000000..3d6c78a --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/reasoning-effort-overuse.md @@ -0,0 +1,38 @@ +--- +id: reasoning-effort-overuse +name: Reasoning Effort Overuse +group: tool-mastery +severity: medium +scope: requests +version: 1 +tags: [model, reasoning, effort, cost] +thresholds: + minSample: 20 + maxRatio: 0.5 +--- + +# Description +Detects sessions where high or maximum reasoning effort is used for the majority of requests on reasoning-capable models. Higher reasoning levels generate many extra "thinking" tokens — every `-high` or `-xhigh` request typically costs 2–4× more output tokens than `-medium` or default for the same answer. + +# When Triggered +{{extra.premiumCount}} of {{extra.totalKnown}} requests with a known reasoning level ({{pct}}) ran at `high` or `max` effort. These extra thinking tokens are billed even when the task doesn't need deep reasoning. + +# How to Improve +Default to `medium` effort and only escalate when the task actually benefits from extra reasoning (complex algorithms, ambiguous spec, multi-step planning). For routine edits, refactors, or factual questions, lower effort delivers the same result for a fraction of the output tokens. + +# Examples +{{normalizeModel(modelId)}} · effort: {{reasoningEffort}} + +# Detection Logic +```detect +scan: requests +match: reasoningEffort == "high" OR reasoningEffort == "max" +aggregate: count +stats: reasoningEffortStats(allReqs, "high") +emitCount: stats.premiumCount +emitTotal: stats.totalKnown +premiumCount: stats.premiumCount +totalKnown: stats.totalKnown +check: stats.totalKnown > thresholds.minSample AND stats.ratio > thresholds.maxRatio +examples: {{normalizeModel(modelId)}} · effort: {{reasoningEffort}} +``` diff --git a/packages/ai-coach-analytics/src/rules/repeated-prompts.md b/packages/ai-coach-analytics/src/rules/repeated-prompts.md new file mode 100644 index 0000000..70ae06c --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/repeated-prompts.md @@ -0,0 +1,38 @@ +--- +id: repeated-prompts +name: Repeated Prompts +group: prompt-quality +severity: medium +scope: requests +version: 1 +tags: [prompt, duplicate, waste] +thresholds: + minDuplicates: 3 + highThreshold: 20 +--- + +# Description +Detects near-duplicate prompts that waste quota without producing new results. + +# When Triggered +{{count}} requests are near-duplicates across {{extra.distinctCount}} distinct prompts. This wastes quota without new results. + +# How to Improve +If a prompt isn't working, rephrase it or provide more context instead of retrying the same message. + +# Examples +"{{message}}..." (repeated {{extra.repeatCount}}x) + +# Detection Logic +```detect +scan: requests +match: messageLength > 0 +aggregate: count +dupes: duplicateGroups(matched, 10, thresholds.minDuplicates) +emitCount: dupes.totalDupes +emitTotal: total +distinctCount: dupes.distinctCount +check: dupes.totalDupes >= thresholds.minDuplicates +severity: dupes.totalDupes > thresholds.highThreshold +examples: "{{messageText | truncate:60}}" +``` diff --git a/packages/ai-coach-analytics/src/rules/runaway-agent-loops.md b/packages/ai-coach-analytics/src/rules/runaway-agent-loops.md new file mode 100644 index 0000000..d398b07 --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/runaway-agent-loops.md @@ -0,0 +1,35 @@ +--- +id: runaway-agent-loops +name: Runaway Agent Loops +group: session-hygiene +severity: high +scope: requests +version: 1 +tags: [session, agent, tools, loops] +thresholds: + minToolsPerReq: 15 + minReqs: 3 +--- + +# Description +Detects agentic requests that use an excessive number of tools per request, indicating the agent may be spinning on failing approaches. + +# When Triggered +{{count}} agentic requests used {{extra.minToolsPerReq}}+ tools each (avg {{extra.avgTools}}). The agent may be spinning on failing approaches. + +# How to Improve +Break complex tasks into smaller, focused requests. If the agent is looping, cancel and rephrase with clearer constraints. + +# Examples +{{extra.toolCount}} tools: "{{message}}..." + +# Detection Logic +```detect +scan: requests +match: toolsUsed.length >= thresholds.minToolsPerReq AND (agentMode == "agent" OR agentName != "") +aggregate: count +totalTools: flatCount(matched, "toolsUsed") +avgTools: round(totalTools / count) +check: count >= thresholds.minReqs +examples: {{toolsUsed.length}} tools: "{{messageText | truncate:60}}" +``` diff --git a/packages/ai-coach-analytics/src/rules/session-drift.md b/packages/ai-coach-analytics/src/rules/session-drift.md new file mode 100644 index 0000000..3468c41 --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/session-drift.md @@ -0,0 +1,34 @@ +--- +id: session-drift +name: Session Drift +group: session-hygiene +severity: medium +scope: sessions +version: 1 +tags: [session, focus, drift] +thresholds: + maxWorkTypes: 4 + minReqsPerSession: 5 + minSessions: 3 +--- + +# Description +Detects sessions that cover too many different task types, confusing the AI context window. + +# When Triggered +{{count}} sessions cover {{extra.maxWorkTypes}}+ different task types. Mixed-purpose sessions confuse the AI context. + +# How to Improve +Start a new session when switching task types (bug fix to feature, docs to testing, etc.). Focused sessions get better responses. + +# Examples +{{extra.workspace}}: {{extra.workTypes}} + +# Detection Logic +```detect +scan: sessions +match: requestCount >= thresholds.minReqsPerSession AND workTypeCount(requests) >= thresholds.maxWorkTypes +aggregate: count +check: count > thresholds.minSessions +examples: {{workspaceName}}: {{workTypeCount(requests)}} work types in one session +``` diff --git a/packages/ai-coach-analytics/src/rules/slow-responses.md b/packages/ai-coach-analytics/src/rules/slow-responses.md new file mode 100644 index 0000000..47183de --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/slow-responses.md @@ -0,0 +1,34 @@ +--- +id: slow-responses +name: Slow Responses +group: session-hygiene +severity: low +scope: requests +version: 1 +tags: [session, performance, slow] +thresholds: + slowMs: 30000 + minCount: 5 +--- + +# Description +Detects requests with unusually long response times, which may indicate overly broad or complex prompts. + +# When Triggered +{{count}} requests took over 30 seconds (avg {{extra.avgSec}}s). May indicate overly broad prompts. + +# How to Improve +Break complex tasks into smaller, focused requests. Use lighter models for simple questions. + +# Examples +{{extra.elapsedSec}}s: "{{message}}..." + +# Detection Logic +```detect +scan: requests +match: totalElapsed > thresholds.slowMs AND totalElapsed > 0 +aggregate: count +avgSec: round(avgField(matched, "totalElapsed") / 1000) +check: count > thresholds.minCount +examples: "{{messageText | truncate:50}}" +``` diff --git a/packages/ai-coach-analytics/src/rules/speed-accept.md b/packages/ai-coach-analytics/src/rules/speed-accept.md new file mode 100644 index 0000000..21183aa --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/speed-accept.md @@ -0,0 +1,41 @@ +--- +id: speed-accept +name: Speed Accept (No Review) +group: code-review +severity: high +scope: sessions +version: 1 +tags: [review, speed, accept] +thresholds: + minAiLoc: 20 + maxGapMs: 15000 + minOccurrences: 5 +--- + +# Description +Detects instances where the next message was sent within seconds of receiving large AI code blocks, indicating no time for review. + +# When Triggered +{{count}} times you sent the next message within {{extra.maxGapSec}}s of receiving {{extra.minAiLoc}}+ lines of AI code (avg {{extra.avgLoc}} LOC, avg {{extra.avgGap}}s gap). Not enough time to review. + +# How to Improve +Take time to read AI-generated code before moving on. Review for correctness, security issues, and edge cases. A quick glance is not a review. + +# Examples +{{extra.workspace}}: {{extra.aiLoc}} AI LOC, {{extra.gapSec}}s gap -> "{{message}}..." + +# Detection Logic +```detect +scan: sessions +match: requestCount >= 2 +aggregate: count +pairs: adjacentPairCount(allSessions, thresholds.minAiLoc, thresholds.maxGapMs) +emitCount: pairs.count +emitTotal: count +avgLoc: pairs.avgLoc +avgGap: pairs.avgGap +maxGapSec: round(thresholds.maxGapMs / 1000) +minAiLoc: thresholds.minAiLoc +check: pairs.count >= thresholds.minOccurrences +examples: {{workspaceName}}: speed-accept detected +``` diff --git a/packages/ai-coach-analytics/src/rules/tunnel-vision.md b/packages/ai-coach-analytics/src/rules/tunnel-vision.md new file mode 100644 index 0000000..fbd853b --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/tunnel-vision.md @@ -0,0 +1,38 @@ +--- +id: tunnel-vision +name: Single-Workspace Tunnel Vision +group: code-review +severity: low +scope: sessions +version: 1 +tags: [review, workspace, diversity] +thresholds: + maxTopRate: 0.95 + minReqs: 50 + minWorkspaces: 3 +--- + +# Description +Detects when the vast majority of requests are concentrated in a single workspace, missing opportunities to use AI across projects. + +# When Triggered +{{pct}} of all requests ({{count}}/{{total}}) are in "{{extra.topWorkspace}}". Copilot can help across all your projects. + +# How to Improve +Try using Copilot in other workspaces too. It can help with documentation, testing, DevOps, and exploratory coding across your entire workflow. + +# Examples +{{extra.workspace}}: {{extra.reqCount}} requests ({{extra.reqPct}}%) + +# Detection Logic +```detect +scan: sessions +match: true +aggregate: count +top: groupTopBySum(allSessions, "workspaceName", "requestCount") +reqTotal: sumField(allSessions, "requestCount") +emitCount: top.sum +emitTotal: reqTotal +topWorkspace: top.key +check: top.share > thresholds.maxTopRate AND top.sum >= thresholds.minReqs AND top.count >= thresholds.minWorkspaces +``` diff --git a/packages/ai-coach-analytics/src/rules/verbose-output.md b/packages/ai-coach-analytics/src/rules/verbose-output.md new file mode 100644 index 0000000..e528cb2 --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/verbose-output.md @@ -0,0 +1,42 @@ +--- +id: verbose-output +name: Verbose Model Output +group: prompt-quality +severity: medium +scope: requests +version: 1 +tags: [tokens, output, verbosity, cost] +thresholds: + minCompletionTokens: 5000 + maxMessageLength: 200 + minSample: 10 + maxRatio: 0.1 +--- + +# Description +Detects requests where the AI generated very long responses (>5K completion tokens) for short, low-context prompts. These rambling outputs burn through completion-token budgets without proportional value. + +# When Triggered +{{count}} requests ({{pct}}) produced >{{extra.minCompletionTokens}} completion tokens from prompts shorter than {{extra.maxMessageLength}} characters. Verbose outputs are a major driver of token spend. + +# How to Improve +Be explicit about response length and format: ask for "a concise answer", "a one-line summary", "no commentary", or specify a maximum number of bullets. Add output constraints to your custom instructions for routine tasks. + +# Examples +"{{messageText | truncate:80}}" → {{completionTokens}} completion tokens + +# Detection Logic +```detect +scan: requests +match: completionTokens > thresholds.minCompletionTokens AND messageLength > 0 AND messageLength < thresholds.maxMessageLength +aggregate: ratio +check: ratio > thresholds.maxRatio AND count > thresholds.minSample +examples: "{{messageText | truncate:80}}" → {{completionTokens}} tokens +``` + +# Tests +```test +{messageText: "fix this", messageLength: 8, completionTokens: 8000} -> triggered +{messageText: "Explain in detail how the OAuth flow works including token refresh, PKCE, and edge cases for mobile clients", messageLength: 110, completionTokens: 8000} -> clean +{messageText: "fix this", messageLength: 8, completionTokens: 200} -> clean +``` diff --git a/packages/ai-coach-analytics/src/rules/verbose-prompt-no-compression.md b/packages/ai-coach-analytics/src/rules/verbose-prompt-no-compression.md new file mode 100644 index 0000000..7d77953 --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/verbose-prompt-no-compression.md @@ -0,0 +1,34 @@ +--- +id: verbose-prompt-no-compression +name: Verbose Prompts Without Compression +group: prompt-quality +severity: low +scope: requests +version: 1 +tags: [prompt, verbosity, tokens, cost] +thresholds: + minMessageLength: 800 + minSample: 15 + maxRatio: 0.2 +--- + +# Description +Detects user prompts that are unusually long and full of low-signal "fluff" words (please, kindly, thanks, basically, essentially, definitely, absolutely, simply, very, quite, somewhat, certainly). Long verbose prompts inflate input tokens on every turn — and where compression skills are available (e.g. caveman/cavecrew), the user is paying the verbosity tax twice (once on the user message, once on the system instructions). + +# When Triggered +{{count}} verbose prompts ({{pct}}) of length ≥{{thresholds.minMessageLength}} chars contained 2+ filler words. These prompts could be rewritten to half the size with no loss of meaning. + +# How to Improve +Be terse and structured. Replace "please could you kindly write a function that basically just adds two numbers" with "write add(a,b)". Use bullet points instead of paragraphs. Drop pleasantries — Copilot doesn't care, and you pay for every token. Consider installing a compression skill like `caveman/cavecrew` to compress sub-agent results. + +# Examples +"{{messageText | truncate:80}}" ({{messageLength}} chars) + +# Detection Logic +```detect +scan: requests +match: messageLength >= thresholds.minMessageLength AND matches(messageText, "(?i)\\b(please|kindly|thanks|thank you|basically|essentially|definitely|absolutely|simply|very|quite|somewhat|certainly|actually|literally)\\b.*\\b(please|kindly|thanks|thank you|basically|essentially|definitely|absolutely|simply|very|quite|somewhat|certainly|actually|literally)\\b") AND hasSkillByPattern(allReqs, "(?i)cavecrew|caveman|compress") == 0 +aggregate: ratio +check: ratio > thresholds.maxRatio AND count > thresholds.minSample +examples: "{{messageText | truncate:80}}" ({{messageLength}} chars) +``` diff --git a/packages/ai-coach-analytics/src/rules/vibe-coding.md b/packages/ai-coach-analytics/src/rules/vibe-coding.md new file mode 100644 index 0000000..e71322e --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/vibe-coding.md @@ -0,0 +1,40 @@ +--- +id: vibe-coding +name: Vibe Coding +group: code-review +severity: high +scope: sessions +version: 1 +tags: [review, vibe, quality] +thresholds: + minAiLoc: 100 + maxUserPrompts: 5 + minSessions: 3 +--- + +# Description +Detects sessions with high AI code output from minimal prompts with no specs, indicating velocity without understanding. + +# When Triggered +{{count}} sessions show vibe-coding patterns: {{extra.totalVibeLoc}} AI LoC generated with minimal prompts, no specs, and minimal review. Velocity without understanding creates knowledge debt. + +# How to Improve +Slow down. Write specs before coding. Review generated code line by line. Understand what the AI produced before moving on. High LoC output without comprehension is technical debt, not productivity. + +# Examples +{{extra.workspace}}: {{extra.aiLoc}} AI LoC in {{extra.messageCount}} messages -- "{{message}}..." + +# Detection Logic +```detect +scan: sessions +match: flatSumField(requests, "aiCode", "loc") >= thresholds.minAiLoc AND \ + requestCount <= thresholds.maxUserPrompts AND NOT (\ + matches(first(requests).messageText, "(?m)^[-*]\\s") OR \ + matches(first(requests).messageText, "(?m)^\\d+[.)]\\s") OR \ + matches(first(requests).messageText, "(?m)^#+\\s") OR \ + matches(first(requests).messageText, "(?i)\\b(requirements?|spec|acceptance criteria|user stories?|given|when|then|should|must)\\b") OR \ + lineCount(first(requests).messageText) >= 4) +aggregate: count +check: count >= thresholds.minSessions +examples: {{workspaceName}}: {{flatSumField(requests, "aiCode", "loc")}} AI LoC in {{requestCount}} messages +``` diff --git a/packages/ai-coach-analytics/src/rules/weekend-overwork.md b/packages/ai-coach-analytics/src/rules/weekend-overwork.md new file mode 100644 index 0000000..18094fa --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/weekend-overwork.md @@ -0,0 +1,33 @@ +--- +id: weekend-overwork +name: Weekend Overwork +group: session-hygiene +severity: low +scope: requests +version: 1 +tags: [session, health, weekend] +thresholds: + maxWeekendRate: 0.25 + minWeekendReqs: 20 +--- + +# Description +Detects a high proportion of requests happening on weekends, which may indicate work-life balance issues. + +# When Triggered +{{count}} requests ({{pct}}) happen on weekends. This may indicate work-life balance issues. + +# How to Improve +Consider maintaining boundaries between work and personal time. Sustained overwork leads to burnout and decreased productivity. + +# Examples +{{extra.weekendReqs}} weekend requests vs {{extra.weekdayReqs}} weekday requests + +# Detection Logic +```detect +scan: requests +match: timestamp > 0 AND (dayOfWeek(timestamp) == 0 OR dayOfWeek(timestamp) == 6) +aggregate: ratio +check: ratio > thresholds.maxWeekendRate AND count > thresholds.minWeekendReqs +examples: "{{messageText | truncate:50}}" +``` diff --git a/packages/ai-coach-analytics/src/rules/yolo-mode.md b/packages/ai-coach-analytics/src/rules/yolo-mode.md new file mode 100644 index 0000000..4271a0d --- /dev/null +++ b/packages/ai-coach-analytics/src/rules/yolo-mode.md @@ -0,0 +1,36 @@ +--- +id: yolo-mode +name: YOLO Mode +group: code-review +severity: high +scope: requests +requiresIdeContext: true +version: 1 +tags: [review, security, auto-approve, unsupervised] +thresholds: + autoApproveRate: 0.9 + minConfirmations: 15 +--- + +# Description +Detects when the vast majority of tool actions are auto-approved, meaning the agent runs virtually unsupervised. + +# When Triggered +{{count}} of {{total}} tool actions ({{pct}}) were auto-approved. The agent is running virtually unsupervised. + +# How to Improve +Disable blanket auto-approve. Review file edits, terminal commands, and web searches individually. Use session-scoped approval only for trusted, low-risk tools. + +# Examples +Auto-approved: {{extra.tools}} + +# Detection Logic +```detect +scan: requests +match: toolConfirmations.length > 0 +aggregate: count +yolo: yoloStats(matched) +emitCount: yolo.autoApproved +emitTotal: yolo.totalConfirmations +check: yolo.ratio > thresholds.autoApproveRate AND yolo.totalConfirmations >= thresholds.minConfirmations +``` diff --git a/packages/ai-coach-analytics/src/schemas.ts b/packages/ai-coach-analytics/src/schemas.ts new file mode 100644 index 0000000..02b332c --- /dev/null +++ b/packages/ai-coach-analytics/src/schemas.ts @@ -0,0 +1,121 @@ +/*--------------------------------------------------------------------------------------------- + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the MIT License. See LICENSE in the project root for license information. + *--------------------------------------------------------------------------------------------*/ + +import { z } from 'zod'; +import type { Session } from './types'; +import { warnCore } from './log'; + +/* ---- Sub-schemas ---- */ + +export const CodeBlockSchema = z.object({ + language: z.string(), + loc: z.number(), +}).passthrough(); + +export const ToolConfirmationSchema = z.object({ + toolId: z.string(), + confirmationType: z.number(), + autoApproveScope: z.string().optional(), + isTerminal: z.boolean(), + commandLine: z.string().optional(), +}).passthrough(); + +export const CompactionEventSchema = z.object({ + mode: z.enum(['full', 'simple']), + numRounds: z.number(), + numRoundsSinceLastSummarization: z.number(), + contextLengthBefore: z.number(), + durationMs: z.number(), + model: z.string(), + outcome: z.string(), +}).passthrough(); + +export const TodoItemSchema = z.object({ + id: z.number(), + title: z.string(), + status: z.enum(['not-started', 'in-progress', 'completed']), +}).passthrough(); + +export const ModelUsageSchema = z.object({ + inputTokens: z.number(), + outputTokens: z.number(), + cacheReadTokens: z.number(), + cacheWriteTokens: z.number(), + reasoningTokens: z.number().optional(), +}).passthrough(); + +/* ---- SessionRequest ---- */ + +export const SessionRequestSchema = z.object({ + requestId: z.string(), + timestamp: z.number().nullable(), + messageText: z.string(), + responseText: z.string(), + isCanceled: z.boolean(), + agentName: z.string(), + agentMode: z.string(), + modelId: z.string(), + toolsUsed: z.array(z.string()), + editedFiles: z.array(z.string()), + referencedFiles: z.array(z.string()), + slashCommand: z.string(), + variableKinds: z.record(z.string(), z.number()), + customInstructions: z.array(z.string()), + skillsUsed: z.array(z.string()), + firstProgress: z.number().nullable(), + totalElapsed: z.number().nullable(), + messageLength: z.number(), + responseLength: z.number(), + userCode: z.array(CodeBlockSchema), + aiCode: z.array(CodeBlockSchema), + toolConfirmations: z.array(ToolConfirmationSchema), + promptTokens: z.number().nullable(), + completionTokens: z.number().nullable(), + cacheReadTokens: z.number().nullable(), + cacheWriteTokens: z.number().nullable(), + compaction: CompactionEventSchema.nullable(), + todoSnapshot: z.array(TodoItemSchema).nullable(), + workType: z.string(), + reasoningEffort: z.enum(['max', 'high', 'medium', 'low']).nullable().optional(), + endState: z.enum(['pending', 'errored', 'no-data']).optional(), +}).passthrough(); + +/* ---- Session ---- */ + +export const SessionSchema = z.object({ + sessionId: z.string(), + workspaceId: z.string(), + workspaceName: z.string(), + location: z.string(), + harness: z.string(), + creationDate: z.number().nullable(), + lastMessageDate: z.number().nullable(), + requestCount: z.number(), + requests: z.array(SessionRequestSchema), + modelUsage: z.record(z.string(), ModelUsageSchema).optional(), + endReason: z.enum(['shutdown', 'active', 'aborted', 'unknown']).optional(), + hasDevcontainer: z.boolean().optional(), + customInstructionsBytes: z.number().optional(), + workspaceRootPath: z.string().optional(), +}).passthrough(); + +/* ---- Validation helper ---- */ + +/** + * Validates parsed sessions and filters out invalid ones. + * Logs warnings for malformed sessions but doesn't throw. + */ +export function validateSessions(sessions: unknown[], source: string): Session[] { + const valid: Session[] = []; + for (const s of sessions) { + const result = SessionSchema.safeParse(s); + if (result.success) { + valid.push(result.data as Session); + } else { + warnCore('schema', `Invalid session from ${source}: ${result.error.issues[0]?.message ?? 'unknown'}`); + } + } + return valid; +} diff --git a/packages/ai-coach-analytics/src/types.ts b/packages/ai-coach-analytics/src/types.ts new file mode 100644 index 0000000..8846138 --- /dev/null +++ b/packages/ai-coach-analytics/src/types.ts @@ -0,0 +1,14 @@ +/*--------------------------------------------------------------------------------------------- + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the MIT License. See LICENSE in the project root for license information. + *--------------------------------------------------------------------------------------------*/ + +export * from './types/session-types'; +export * from './types/analytics-types'; +export * from './types/catalog-types'; +export * from './types/insights-types'; +export * from './types/config-types'; +export * from './types/context-types'; +export * from './types/rule-types'; +// rpc-types intentionally omitted: it pulls in analyzer-images.ts (image gallery +// RPC plumbing) which is out of scope for this package's parser/rule-engine slice. diff --git a/packages/ai-coach-analytics/src/types/analytics-types.ts b/packages/ai-coach-analytics/src/types/analytics-types.ts new file mode 100644 index 0000000..c05952b --- /dev/null +++ b/packages/ai-coach-analytics/src/types/analytics-types.ts @@ -0,0 +1,653 @@ +/*--------------------------------------------------------------------------------------------- + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the MIT License. See LICENSE in the project root for license information. + *--------------------------------------------------------------------------------------------*/ + +export interface DailyHarnessData { + harness: string; + requests: number[]; + sessions: number[]; + loc: number[]; +} + +export interface DailyActivity { + labels: string[]; + values: number[]; + loc: number[]; + sessions: number[]; + workspaces: number[]; + byHarness: DailyHarnessData[]; +} + +export interface HourlyDistribution { + hours: number[]; + byType: Record; +} + +export interface HeatmapData { + heatmap: number[][]; + byType: Record; + /** 7×24 grid of focus scores (0-100) based on activity density */ + focusHeatmap: number[][]; +} + +export interface CalendarDay { + date: string; + requests: number; + focusScore: number; + dow: number; +} + +export interface CalendarActivityData { + days: CalendarDay[]; + maxRequests: number; +} + +export interface CodeProductionData { + summary: { + totalAiLoc: number; + totalUserLoc: number; + totalLoc: number; + aiBlocks: number; + userBlocks: number; + aiRatio: number; + locCost2010: number; + costPerLoc: number; + }; + byLanguage: { + labels: string[]; + aiLoc: number[]; + userLoc: number[]; + }; + dailyTimeline: { + labels: string[]; + aiLoc: number[]; + userLoc: number[]; + }; + byWorkspace: { + labels: string[]; + aiLoc: number[]; + userLoc: number[]; + }; + dailyByWorkspace: Record; + dailyByModel: Record; + dailyByHarness: Record; +} + +export interface ConsumptionData { + totalRequests: number; + avgPerDay: number; + avgPerWeek: number; + avgPerMonth: number; + modelTotals: Record; + defaultMultipliers: Record; + daily: { labels: string[]; values: number[]; byModel: Record }; + weekly: { labels: string[]; values: number[]; byModel: Record }; + monthly: { labels: string[]; values: number[]; byModel: Record }; +} + +export interface TimelineSession { + sessionId: string; + workspaceName: string; + sessionName: string; + firstActivity: number; + lastActivity: number; + requestCount: number; + totalRequestCount: number; + requests: TimelineRequest[]; +} + +export interface TimelineRequest { + timestamp: number; + messageText: string; + responseText: string; + messageLength: number; + responseLength: number; + agentName: string; + modelId: string; + toolsUsed: string[]; + editedFiles: string[]; + referencedFiles: string[]; + preview: string; + loc?: number; + workType?: string; +} + +export interface DayTimeline { + date: string; + mode: string; + rangeLabel: string; + dayStart: number; + dayEnd: number; + sessions: TimelineSession[]; + sessionCount: number; + maxConcurrent: number; + prevDay: string | null; + nextDay: string | null; + firstDay: string | null; + activeDates: { date: string; count: number }[]; +} + +export interface SessionListItem { + sessionId: string; + workspaceName: string; + workspaceId: string; + creationDate: number | null; + lastMessageDate: number | null; + requestCount: number; + firstMessage: string; +} + +export interface SessionList { + total: number; + page: number; + pageSize: number; + sessions: SessionListItem[]; +} + +export interface WorkspaceBreakdown { + labels: string[]; + values: number[]; +} + +export interface BurndownConfig { + sku: string; + customBudget?: number; + modelBudgets?: Record; + month?: string; +} + +export interface BurndownData { + currentMonth: string; + daysInMonth: number; + dayOfMonth: number; + budget: number; + consumed: number; + projected: number; + dailyConsumption: { labels: string[]; values: number[]; cumulative: number[] }; + projectedLine: number[]; + budgetLine: number[]; + status: 'on-track' | 'warning' | 'over-budget'; + recommendation: string; +} + +/* ---- AI Credit (usage-based billing) types ---- */ + +export interface AiCreditModelBreakdown { + requests: number; + /** Number of requests included in token/credit math (with both native input + output, or session-aggregated). */ + countedRequests: number; + /** Output-only requests (output tokens captured, input not) — billed lower-bound, surfaced separately. */ + partialRequests: number; + /** Requests in still-active or aborted-pre-response sessions (token data not yet finalized). + * Excluded from `missingPct` denominator — they may still arrive. */ + pendingRequests: number; + /** Requests where the harness/source structurally cannot record token data (Xcode, CLI abort-only, etc.). + * Permanent and excluded from `missingPct` denominator. */ + noDataRequests: number; + /** @deprecated No longer tracked (Claude harness unified). Always 0. */ + delegatedRequests: number; + /** Uncached input tokens (subset of `inputTokens`) — used for input-rate billing math. */ + uncachedInputTokens: number; + /** Total input tokens (uncached + cached). */ + inputTokens: number; + outputTokens: number; + cacheReadTokens: number; + cacheWriteTokens: number; + credits: number; + /** % of *finalizable* requests for this model with no token data (denominator excludes pending + no-data, same as Token Coverage). */ + missingPct: number; + /** Number of finalizable requests (`requests - pendingRequests - noDataRequests`). + * When this is 0, `missingPct` is 0 by convention but the UI should render "N/A" — there's + * nothing in this slice that *could* have token data. */ + finalizableRequests: number; + /** Harness names that contributed requests for this model (sorted). */ + harnesses: string[]; +} + +export interface AiCreditData { + totalCredits: number; + totalInputTokens: number; + totalOutputTokens: number; + totalCacheReadTokens: number; + totalCacheWriteTokens: number; + /** Total request count in the filter window (includes those with missing token data) */ + totalRequests: number; + /** Number of requests with complete billing data (counted in totals/credits — includes session-aggregated CLI requests) */ + countedRequests: number; + /** Output-only requests (see `AiCreditModelBreakdown.partialRequests`). */ + partialRequests: number; + /** In-flight / aborted-pre-response requests (excluded from `missingPct`). */ + pendingRequests: number; + /** Permanently untracked requests (excluded from `missingPct`). */ + noDataRequests: number; + /** @deprecated No longer tracked (Claude harness unified). Always 0. */ + delegatedRequests: number; + /** Finalizable requests = `totalRequests - pendingRequests - noDataRequests`. */ + finalizableRequests: number; + /** % of *finalizable* requests with no token data (matches Token Coverage semantics). */ + missingPct: number; + avgCreditsPerRequest: number; + avgCreditsPerDay: number; + costByModel: Record; + daily: { labels: string[]; credits: number[]; cumulative: number[]; byModel: Record }; + weekly: { labels: string[]; credits: number[]; cumulative: number[]; byModel: Record }; + /** Daily total-token (input + output) consumption broken down by workspace. + * `labels` matches `daily.labels`; each workspace key maps to an aligned array of token totals. */ + dailyTokensByWorkspace: { labels: string[]; byWorkspace: Record }; + /** Daily total-token (input + output) consumption broken down by harness. + * `labels` matches `daily.labels`; each harness key maps to an aligned array of token totals. */ + dailyTokensByHarness: { labels: string[]; byHarness: Record }; + topRequests: Array<{ + timestamp: number; + model: string; + inputTokens: number; + outputTokens: number; + credits: number; + /** Per-request token-data classification (matches RequestBilling.status). */ + status: 'complete' | 'partial' | 'pending' | 'no-data' | 'missing'; + /** "exact" = per-request native data; "session-aggregated" = derived from session-level totals (CLI). */ + aggregationKind: 'exact' | 'session-aggregated'; + preview: string; + workspace: string; + harness: string; + fullPrompt: string; + }>; +} + +/** Token-data coverage breakdown — surfaces how many sessions/requests + * per harness and per workspace have native token data available. Used by + * the "Token Coverage" overview to diagnose data quality. */ +export interface TokenCoverageData { + totalSessions: number; + totalRequests: number; + totalWorkspaces: number; + totalHarnesses: number; + /** Sessions where every counted request has billing-complete tokens. */ + countedSessions: number; + /** Requests with billing-complete data (per-request native input + output OR session-level aggregated). */ + countedRequests: number; + /** Requests genuinely missing token data — excludes pending, partial, and no-data. */ + missingRequests: number; + /** Requests where we have output tokens but no input (e.g. VS Code copilot/auto). */ + partialRequests: number; + /** Requests in active or aborted sessions where token data was never finalized. */ + pendingRequests: number; + /** Requests that finalized but the harness/source recorded no token usage + * — Xcode (no token capture in the source DB), some 2026-04 VS Code + * copilot/auto and copilot/gpt-5.4 requests, and Copilot CLI turns + * aborted before any model output. Permanent: there's nothing to recover. + * Excluded from the missing% denominator and surfaced separately. */ + noDataRequests: number; + /** Sessions still running (no shutdown event seen). */ + activeSessions: number; + /** Sessions where the user aborted before any model response, so no tokens exist. */ + abortedSessions: number; + /** % of finalizable requests with no billing data. Excludes pending and no-data. */ + missingPct: number; + totalInputTokens: number; + totalOutputTokens: number; + /** Output tokens captured for partial (output-only) requests, kept separate + * from `totalOutputTokens` so the latter only reflects billing-complete totals. */ + partialOutputTokens: number; + totalCacheReadTokens: number; + totalCacheWriteTokens: number; + byHarness: TokenCoverageHarness[]; + byWorkspace: TokenCoverageWorkspace[]; + bySession: TokenCoverageSession[]; + /** Per-month per-harness coverage trend so the user can see when each + * harness started reporting reliable token data. */ + timeline: TokenCoverageTimeline; +} + +export interface TokenCoverageHarness { + harness: string; + sessions: number; + workspaces: number; + requests: number; + countedRequests: number; + partialRequests: number; + pendingRequests: number; + /** Requests permanently without token data (harness limitation or + * finalized-but-not-recorded). Excluded from missingPct denominator. */ + noDataRequests: number; + missingPct: number; + inputTokens: number; + outputTokens: number; + partialOutputTokens: number; + cacheReadTokens: number; + cacheWriteTokens: number; + /** How requests in this harness are billed: "per-request" (Claude/OpenCode/VS Code), + * "session-aggregated" (Copilot CLI shutdown.modelMetrics), or "mixed". */ + source: 'per-request' | 'session-aggregated' | 'mixed' | 'none'; +} + +export interface TokenCoverageWorkspace { + workspaceId: string; + workspaceName: string; + harnesses: string[]; + sessions: number; + requests: number; + countedRequests: number; + partialRequests: number; + pendingRequests: number; + noDataRequests: number; + missingPct: number; + inputTokens: number; + outputTokens: number; +} + +/** Per-session coverage row used by the drill-down table in the + * "Token Coverage" overview. Surfaces which individual sessions are + * missing token data so the user can target their reindexing or + * parser investigation. */ +export interface TokenCoverageSession { + sessionId: string; + harness: string; + workspaceId: string; + workspaceName: string; + firstMessageDate: number | null; + lastMessageDate: number | null; + requests: number; + countedRequests: number; + partialRequests: number; + pendingRequests: number; + noDataRequests: number; + missingPct: number; + inputTokens: number; + outputTokens: number; + /** Session disposition. 'shutdown' = closed cleanly, 'active' = still open, + * 'aborted' = user cancelled before any model response, 'unknown' = harness + * doesn't expose disposition. Drives pending vs missing classification. */ + endReason: 'shutdown' | 'active' | 'aborted' | 'unknown'; + /** How this session's tokens are sourced — same vocabulary as + * TokenCoverageHarness.source, applied to a single session. */ + source: 'per-request' | 'session-aggregated' | 'mixed' | 'none'; +} + +/** Monthly per-harness token coverage timeline. The user can scan this to + * answer "since when does harness X actually report tokens?". Months are + * ISO `YYYY-MM` strings, sorted ascending. */ +export interface TokenCoverageTimeline { + months: string[]; + harnesses: string[]; + /** `cells[harness][month]` — undefined when the harness had no requests + * in that bucket. */ + cells: Record>; +} + +export interface TokenCoverageTimelineCell { + requests: number; + countedRequests: number; + partialRequests: number; + pendingRequests: number; + noDataRequests: number; + /** Missing % over finalizable requests in this bucket (excludes pending and no-data). */ + missingPct: number; +} + +export interface AiCreditBurndownData { + currentMonth: string; + daysInMonth: number; + dayOfMonth: number; + budget: number; + consumed: number; + projected: number; + dailyConsumption: { labels: string[]; values: number[]; cumulative: number[] }; + projectedLine: number[]; + budgetLine: number[]; + /** Status semantics: + * on-track / warning / will-exceed — billing data exists, projection meaningful. + * no-data — no requests at all in this period (or 100% structurally untracked). + * pending-only — requests exist but all are still in-flight; nothing finalized yet. */ + status: 'on-track' | 'warning' | 'will-exceed' | 'no-data' | 'pending-only'; + daysUntilExhaustion: number | null; + safeDailyBudget: number; + projectedOverage: number; + recommendation: string; + /** % of *finalizable* requests (excluding pending + no-data) with no token data. */ + missingPct: number; + /** Total request count in the period (regardless of token availability) */ + totalRequests: number; + /** Requests with complete native token data (input + output) */ + countedRequests: number; + /** Output-only requests in the period. */ + partialRequests: number; + /** In-flight / aborted-pre-response requests in the period. */ + pendingRequests: number; + /** Permanently untracked requests in the period. */ + noDataRequests: number; + /** @deprecated No longer tracked (Claude harness unified). Always 0. */ + delegatedRequests: number; + /** Finalizable requests = total - pending - no-data. */ + finalizableRequests: number; + /** Per-day request counts in each coverage bucket (length = daysInMonth, index 0 = day 1). + * Drives the coverage strip in the Burndown view so users can see *where* in the month + * data is missing. Future days (after `dayOfMonth`) are zero across the board. */ + coverageByDay: { + complete: number[]; + partial: number[]; + pending: number[]; + noData: number[]; + /** @deprecated No longer tracked. Always zeros. */ + delegated: number[]; + missing: number[]; + }; + /** Per-model cumulative token consumption for the month. Each entry has daily cumulative values (length = daysInMonth). */ + byModel: Record; +} + +export interface HarnessComparisonItem { + harness: string; + sessions: number; + requests: number; + avgRequestsPerSession: number; + totalAiLoc: number; + avgResponseLength: number; + topModels: { name: string; count: number }[]; + topTools: { name: string; count: number }[]; + avgElapsed: number | null; + cancelRate: number; + activeDays: number; + firstSeen: string | null; + lastSeen: string | null; +} + +export interface HarnessComparisonData { + harnesses: HarnessComparisonItem[]; + dailyByHarness: { labels: string[]; series: Record }; +} + +/* ---- Parser field-population matrix ---- */ + +export type FieldCategory = 'core' | 'enrichment' | 'auto-computed'; + +export interface FieldMeta { + name: string; + label: string; + category: FieldCategory; +} + +export interface FieldPopulationCell { + populated: number; + total: number; +} + +export interface ParserCoverageData { + fields: FieldMeta[]; + harnesses: string[]; + /** matrix[fieldName][harness] → { populated, total } */ + matrix: Record>; +} + +/** Preview of a representative sample request per harness showing extracted vs missing data */ +export interface ParserPreviewSample { + harness: string; + sessionId: string; + workspaceName: string; + requestIndex: number; + /** Per-field breakdown: field name → { value (stringified), populated } */ + fields: Record; + /** Total fields populated out of total tracked */ + populatedCount: number; + totalFields: number; +} + +export interface ParserPreviewData { + samples: ParserPreviewSample[]; + fields: FieldMeta[]; +} + +export type WorkType = 'feature' | 'bug fix' | 'refactor' | 'code review' | 'docs' | 'test' | 'style' | 'config' | 'other'; + +export const WORK_TYPES: WorkType[] = ['feature', 'bug fix', 'refactor', 'code review', 'docs', 'test', 'style', 'config', 'other']; + +export const WORK_TYPE_COLORS: Record = { + 'feature': '#58a6ff', + 'bug fix': '#f85149', + 'refactor': '#d29922', + 'code review': '#da7756', + 'docs': '#3fb950', + 'test': '#bc8cff', + 'style': '#f778ba', + 'config': '#79c0ff', + 'other': '#8b949e', +}; + +export const SKU_BUDGETS: Record = { + 'pro': 300, + 'pro-plus': 1500, + 'business': 300, + 'enterprise': 1000, +}; + +export interface RecommendationResult { + checkId: string; + name: string; + category: string; + score: number; + status: 'good' | 'needs-improvement' | 'critical'; + finding: string; + recommendation: string; + details?: Record; +} + +export type PracticeGroup = 'prompt-quality' | 'session-hygiene' | 'code-review' | 'tool-mastery' | 'context-management'; + +export const PRACTICE_GROUPS: Record = { + 'prompt-quality': 'Prompt Quality', + 'session-hygiene': 'Session Hygiene', + 'code-review': 'Code Review', + 'tool-mastery': 'Tool Mastery', + 'context-management': 'Context Management', +}; + +export interface OccurrenceDetail { + timestamp: number; + workspace: string; + sessionId: string; + message: string; + model: string; + /** 'workspace' = workspace-level finding, 'session' = session-level, default = request-level */ + kind?: 'workspace' | 'session' | 'request'; + /** Extra numeric stats for workspace-level findings (e.g. codeLoc, mdLoc) */ + stats?: Record; +} + +export interface AntiPattern { + id: string; + name: string; + severity: 'high' | 'medium' | 'low'; + group: PracticeGroup; + occurrences: number; + description: string; + suggestion: string; + examples: string[]; + /** Rich per-occurrence details for timeline + session drilldown */ + details: OccurrenceDetail[]; + /** Weekly histogram: labels (ISO weeks) and counts per week */ + weeklyHist: { labels: string[]; counts: number[] }; +} + +export interface GroupScore { + group: PracticeGroup; + score: number; + wowPct: number; + momPct: number; + topIssue: string | null; + improvements: string[]; + patternCount: number; +} + +export interface AntiPatternData { + patterns: AntiPattern[]; + totalOccurrences: number; + weeklyTrend: { labels: string[]; counts: number[] }; + groupScores: GroupScore[]; + weeklyScores: { labels: string[]; series: { group: PracticeGroup; scores: number[] }[] }; +} + +export interface WorkLifeBalanceResult { + score: number; + totalRequests: number; + weekdayReqs: number; + weekendReqs: number; + weekendRatio: number; + timeDistribution: { lateNight: number; earlyMorning: number; workHours: number; evening: number }; + hours: number[]; + weekdayHours: number[]; + weekendHours: number[]; + avgStartHour: number; + avgEndHour: number; + avgSpanHours: number; + maxStreak: number; + maxBreak: number; + activeDays: number; + weeklyTrend: { labels: string[]; weekday: number[]; weekend: number[] }; +} + +export interface ProjectOverviewItem { + workspaceName: string; + workspaceId: string; + totalRequests: number; + estimatedHours: number; + languages: string[]; + timePattern: string; + topFiles: string[]; + estimatedLoc: number; + gitPath: string | null; +} + +export interface ProjectOverviewData { + projects: ProjectOverviewItem[]; +} + +export interface StatsResult { + totalSessions: number; + totalWorkspaces: number; + totalRequests: number; +} + +export interface WorkflowCluster { + id: string; + label: string; + canonicalPrompt: string; + occurrences: number; + sessions: number; + workspaces: string[]; + harnesses: string[]; + avgCorrectionTurns: number; + totalTurns: number; + cancelRate: number; + firstSeen: string | null; + lastSeen: string | null; + examples: string[]; + skillDraft: string; +} + +export interface WorkflowOptimizationData { + clusters: WorkflowCluster[]; + totalRepetitions: number; + estimatedTimeSavedMins: number; + topWorkspaces: { name: string; clusters: number }[]; +} diff --git a/packages/ai-coach-analytics/src/types/catalog-types.ts b/packages/ai-coach-analytics/src/types/catalog-types.ts new file mode 100644 index 0000000..ec5fe1b --- /dev/null +++ b/packages/ai-coach-analytics/src/types/catalog-types.ts @@ -0,0 +1,43 @@ +/*--------------------------------------------------------------------------------------------- + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the MIT License. See LICENSE in the project root for license information. + *--------------------------------------------------------------------------------------------*/ + +/* ---- AI Skill Triage ---- */ +export type TriageVerdict = 'strong' | 'maybe' | 'skip'; + +export interface TriagedCluster { + id: string; + label: string; + verdict: TriageVerdict; + reason: string; + suggestedSkillName: string | null; +} + +export interface SkillTriageResult { + triaged: TriagedCluster[]; +} + +/* ---- Awesome Copilot Catalog ---- */ +export type CatalogItemKind = 'skill' | 'agent' | 'instruction' | 'hook'; + +export interface CatalogItem { + kind: CatalogItemKind; + id: string; + title: string; + description: string; + category: string; + path: string; + url: string; + relevanceScore: number; + matchReasons: string[]; +} + +export interface CatalogDiscoverResult { + items: CatalogItem[]; + totalScanned: number; +} + +export interface CatalogTriageResult { + items: CatalogItem[]; +} diff --git a/packages/ai-coach-analytics/src/types/config-types.ts b/packages/ai-coach-analytics/src/types/config-types.ts new file mode 100644 index 0000000..ca1f4c4 --- /dev/null +++ b/packages/ai-coach-analytics/src/types/config-types.ts @@ -0,0 +1,85 @@ +/*--------------------------------------------------------------------------------------------- + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the MIT License. See LICENSE in the project root for license information. + *--------------------------------------------------------------------------------------------*/ + +import type { AntiPattern } from './analytics-types'; +import type { AgenticReadinessScore } from './context-types'; + +/* ---- Context Health ---- */ + +export interface ConfigFileInfo { + relativePath: string; + kind: 'instruction' | 'prompt' | 'agent' | 'skill' | 'hook-config' | 'claude-md' | 'other'; + lines: number; + chars: number; + isMarkdown: boolean; + markdownIssues: string[]; + sizeVerdict: 'compact' | 'moderate' | 'oversized'; + /** Epoch ms of file's last modification time */ + lastModified: number | null; +} + +export interface HookCoverageInfo { + hasPreToolUse: boolean; + hasPostToolUse: boolean; + hasSessionStart: boolean; + hasPermissionRequest: boolean; + totalHooks: number; + hookEvents: string[]; +} + +export interface WorkspaceConfigHealth { + workspaceId: string; + workspaceName: string; + rootPath: string; + harness: string; + configFiles: ConfigFileInfo[]; + hasInstructions: boolean; + hasPrompts: boolean; + hasAgents: boolean; + hasSkills: boolean; + hasHooks: boolean; + progressiveDisclosureScore: number; + instructionQualityScore: number; + hookCoverage: HookCoverageInfo | null; + suggestions: string[]; + /** Activity metrics for this workspace within the date range */ + sessionCount: number; + requestCount: number; + lastActivity: number | null; + /** True if context files haven't been updated since last workspace activity */ + staleContext: boolean; + staleDays: number | null; +} + +export interface ContextProvisionScore { + harness: string; + totalRequests: number; + withFileRefs: number; + withCustomInstructions: number; + withSkills: number; + withTools: number; + avgContextItems: number; + score: number; + /** Extended detail fields for the expanded view */ + totalSessions: number; + avgRequestsPerSession: number; + topModels: { model: string; count: number }[]; + topTools: { tool: string; count: number }[]; + topReferencedFiles: { file: string; count: number }[]; + avgPromptLength: number; + cancelRate: number; + agentModeRate: number; + modeDistribution: { mode: string; count: number }[]; + avgResponseLength: number; +} + +export interface ConfigHealthData { + workspaces: WorkspaceConfigHealth[]; + overallScore: number; + agenticReadiness: AgenticReadinessScore; + contextProvisionByHarness: Record; + suggestions: string[]; + contextAntiPatterns: AntiPattern[]; +} diff --git a/packages/ai-coach-analytics/src/types/context-types.ts b/packages/ai-coach-analytics/src/types/context-types.ts new file mode 100644 index 0000000..de7ddb7 --- /dev/null +++ b/packages/ai-coach-analytics/src/types/context-types.ts @@ -0,0 +1,219 @@ +/*--------------------------------------------------------------------------------------------- + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the MIT License. See LICENSE in the project root for license information. + *--------------------------------------------------------------------------------------------*/ + +import type { AntiPattern } from './analytics-types'; + +/* ---- Context Management (token-level health) ---- */ + +export interface ContextVerdictThresholds { + optimalUtilization: number; + limitedUtilization: number; + adaptive: boolean; + sampleSize: number; +} + +export interface WorkspaceContextScore { + workspaceId: string; + workspaceName: string; + harness: string; + sessionCount: number; + requestsWithTokens: number; + avgPromptTokens: number; + maxPromptTokens: number; + /** Percentage of estimated context window used on average (0-100) */ + avgUtilization: number; + /** Percentage of estimated context window used at peak (0-100) */ + peakUtilization: number; + /** Percentage of requests that are at or above the saturation threshold (60% utilization) */ + saturation: number; + compactionCount: number; + /** Reserved for compatibility; context views no longer surface cost efficiency */ + costEfficiency: number | null; + /** Composite score 0-100 — higher is better context hygiene */ + score: number; + verdict: 'optimal' | 'degraded' | 'limited'; +} + +export interface ContextTrend { + label: string; // week label + avgUtilization: number; + compactions: number; + sessionsOverThreshold: number; +} + +/** Per-workspace weekly utilization data for the trend chart "per-workspace" view */ +export interface WorkspaceTrendSeries { + workspaceName: string; + /** Sparse array aligned to the same week labels as ContextTrend[], null = no data that week */ + data: (number | null)[]; +} + +/** A timeline event that occurred during a session (compaction, todo change, etc.) */ +export interface SessionTimelineEvent { + /** 0-based request index within the session */ + requestIndex: number; + type: 'compaction' | 'todo-add' | 'todo-progress' | 'todo-complete'; + label: string; +} + +/** Per-session context detail returned when drilling into a workspace */ +export interface SessionContextDetail { + sessionId: string; + date: string; + harness: string; + requestCount: number; + requestsWithTokens: number; + avgPromptTokens: number; + maxPromptTokens: number; + avgUtilization: number; + peakUtilization: number; + contextWindow: number; + compactionCount: number; + /** Token counts per request (for sparkline / token curve). `null` for + * requests without native token data — keeps the request index aligned + * with `events` and `requestQueries` while telling the chart to draw a + * gap rather than a misleading 0 dip. */ + tokenCurve: (number | null)[]; + /** Percentage of requests at or above the saturation threshold */ + saturation: number; + /** Reserved for compatibility; context views no longer surface cost efficiency */ + costEfficiency: number | null; + verdict: 'optimal' | 'degraded' | 'limited'; + /** Timeline events (compactions, todo additions/completions) indexed by request */ + events: SessionTimelineEvent[]; + /** Truncated user query per request (aligned with tokenCurve) for tooltip display */ + requestQueries: string[]; + /** True when per-request token data is available (enables token curve chart). + * False for harnesses that only report session-aggregated tokens (e.g. Copilot CLI). */ + hasPerRequestTokens: boolean; +} + +export interface WorkspaceContextSessionsData { + workspaceName: string; + estimatedContextWindow: number; + thresholds: ContextVerdictThresholds; + sessions: SessionContextDetail[]; +} + +export interface ContextManagementData { + /** Overall context health score 0-100 */ + overallScore: number; + /** Estimated context window size (tokens) based on observed maximums */ + estimatedContextWindow: number; + /** Verdict cutoffs used for this filtered dataset */ + thresholds: ContextVerdictThresholds; + /** Per-workspace breakdown */ + workspaces: WorkspaceContextScore[]; + /** Weekly trend data */ + trend: ContextTrend[]; + /** Per-workspace weekly trend (top 10 by request volume) */ + workspaceTrend: WorkspaceTrendSeries[]; + /** Total compaction events observed */ + totalCompactions: number; + /** Full (deep summarization) compaction events — automated context management */ + fullCompactions: number; + /** Simple (lightweight truncation) compaction events */ + simpleCompactions: number; + /** Sessions with token data available */ + sessionsWithTokenData: number; + totalSessions: number; + /** Anti-patterns specific to context management */ + antiPatterns: AntiPattern[]; + /** Actionable tips based on observed patterns */ + tips: string[]; +} + +/* ---- Context Review (agent-based) ---- */ + +export type ContextReviewCategory = + | 'clarity' // Is the intent clear to an AI agent? + | 'specificity' // Are constraints, tech stack, coding standards defined? + | 'structure' // Headings, sections, frontmatter, logical ordering + | 'completeness' // Missing typical sections (project overview, conventions, testing, etc.) + | 'staleness' // Outdated references, deprecated patterns + | 'redundancy' // Duplicated info across files, conflicting instructions + | 'actionability'; // Can the AI act on these instructions or are they too vague? + +export interface ContextReviewFinding { + category: ContextReviewCategory; + severity: 'good' | 'warning' | 'critical'; + file: string; + finding: string; + suggestion: string; +} + +export interface ContextReviewResult { + workspaceId: string; + workspaceName: string; + overallGrade: 'A' | 'B' | 'C' | 'D' | 'F'; + overallScore: number; // 0-100 + categoryScores: Record; // 0-100 each + findings: ContextReviewFinding[]; + summary: string; +} + +/* ---- Agentic Readiness Score ---- */ + +export interface AgenticReadinessScore { + score: number; // 0-100 + signals: AgenticReadinessSignal[]; +} + +export interface AgenticReadinessSignal { + id: string; + label: string; + present: boolean; + weight: number; + detail: string; +} + +/* ---- Flow State Analysis ---- */ + +export interface FlowSession { + sessionId: string; + workspaceName: string; + harness: string; + date: string; + startTs: number; + endTs: number; + durationMin: number; + requestCount: number; + /** Median follow-up latency in seconds (time from response to next prompt) */ + medianFollowUpSec: number; + /** Under 30s follow-up rate — fraction of gaps where user replied within 30s */ + rapidFollowUpRate: number; + /** 0-100 flow score for this session */ + flowScore: number; + flowLabel: 'deep' | 'moderate' | 'shallow' | 'fragmented'; +} + +export interface FlowDay { + date: string; + /** Longest uninterrupted work block in minutes */ + longestBlockMin: number; + /** Total continuous hours on projects */ + totalHours: number; + /** Number of distinct work blocks (gap > 15 min = new block) */ + blockCount: number; + /** Average flow score across sessions that day */ + avgFlowScore: number; + flowLabel: 'deep' | 'moderate' | 'shallow' | 'fragmented'; + sessions: FlowSession[]; +} + +export interface FlowStateData { + days: FlowDay[]; + overallFlowScore: number; + avgFollowUpSec: number; + avgBlockMin: number; + deepFlowDays: number; + totalDays: number; + /** Weekly trend: labels + avg flow scores */ + weeklyTrend: { labels: string[]; scores: number[] }; + /** Per-hour flow score distribution (24 slots) — best hours for deep work */ + hourlyFlow: number[]; + /** Top suggestions */ + suggestions: string[]; +} diff --git a/packages/ai-coach-analytics/src/types/insights-types.ts b/packages/ai-coach-analytics/src/types/insights-types.ts new file mode 100644 index 0000000..c050fc6 --- /dev/null +++ b/packages/ai-coach-analytics/src/types/insights-types.ts @@ -0,0 +1,103 @@ +/*--------------------------------------------------------------------------------------------- + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the MIT License. See LICENSE in the project root for license information. + *--------------------------------------------------------------------------------------------*/ + +/* ---- Session Intent Classification ---- */ +export type SessionIntent = 'Planning' | 'Implementation' | 'Debugging' | 'Review' | 'Exploration'; + +export const SESSION_INTENTS: SessionIntent[] = ['Planning', 'Implementation', 'Debugging', 'Review', 'Exploration']; + +export const INTENT_COLORS: Record = { + 'Planning': '#58a6ff', + 'Implementation': '#3fb950', + 'Debugging': '#f85149', + 'Review': '#d29922', + 'Exploration': '#bc8cff', +}; + +/* ---- Learning Velocity ---- */ +export interface LearningVelocityData { + weeklyLanguages: { week: string; languages: string[]; newLanguages: string[] }[]; + totalLanguagesEncountered: number; + totalNewLanguagesLearned: number; + velocityTrend: { labels: string[]; newLanguages: number[]; cumulativeLanguages: number[] }; + topLanguages: { language: string; firstSeen: string; weekCount: number }[]; +} + +/* ---- Intent Classification ---- */ +export interface IntentClassificationData { + sessionIntents: { sessionId: string; workspaceName: string; intent: SessionIntent; requestCount: number; date: string }[]; + distribution: Record; + weeklyDistribution: { labels: string[]; series: Record }; + avgRequestsByIntent: Record; +} + +/* ---- Spec-Driven Flag ---- */ +export interface SpecDrivenData { + totalSessions: number; + specDrivenCount: number; + specDrivenRate: number; + weeklyTrend: { labels: string[]; specDriven: number[]; unstructured: number[] }; + unstructuredExamples: { workspaceName: string; firstPrompt: string; date: string }[]; +} + +/* ---- Production vs Review Ratio ---- */ +export interface ProductionReviewData { + totalAiLoc: number; + estimatedReviewedLoc: number; + reviewRatio: number; + weeklyTrend: { labels: string[]; produced: number[]; estimated_reviewed: number[] }; + sessionsWithoutReview: number; + avgReviewGapSec: number; +} + +/* ---- Sustainable Pace ---- */ +export interface SustainablePaceData { + weeklyTrend: { + labels: string[]; + lateNightReqs: number[]; + weekendReqs: number[]; + totalReqs: number[]; + avgSessionLength: number[]; + }; + burnoutRisk: 'low' | 'medium' | 'high'; + alerts: string[]; + currentStreak: number; + weekendTrending: 'stable' | 'increasing' | 'decreasing'; + lateNightTrending: 'stable' | 'increasing' | 'decreasing'; +} + +/* ---- Prompt Engineering Maturity ---- */ +export interface PromptMaturityData { + overallGrade: 'A' | 'B' | 'C' | 'D' | 'F'; + score: number; + dimensions: { + constraints: number; + successCriteria: number; + verificationSteps: number; + contextProvision: number; + specificity: number; + }; + weeklyTrend: { labels: string[]; scores: number[] }; + samplePrompts: { text: string; grade: string; issues: string[] }[]; +} + +/* ---- Migration Readiness ---- */ +export interface MigrationReadinessData { + primaryHarness: string; + missingFeatures: { feature: string; availableIn: string[]; description: string }[]; + readinessScore: number; + featureUsage: { feature: string; used: boolean; harnesses: string[] }[]; +} + +/* ---- Insights aggregate ---- */ +export interface InsightsData { + learningVelocity: LearningVelocityData; + intentClassification: IntentClassificationData; + specDriven: SpecDrivenData; + productionReview: ProductionReviewData; + sustainablePace: SustainablePaceData; + promptMaturity: PromptMaturityData; + migrationReadiness: MigrationReadinessData; +} diff --git a/packages/ai-coach-analytics/src/types/rule-types.ts b/packages/ai-coach-analytics/src/types/rule-types.ts new file mode 100644 index 0000000..27613fc --- /dev/null +++ b/packages/ai-coach-analytics/src/types/rule-types.ts @@ -0,0 +1,218 @@ +/*--------------------------------------------------------------------------------------------- + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the MIT License. See LICENSE in the project root for license information. + *--------------------------------------------------------------------------------------------*/ + +/* Types for the data-driven rule engine that powers anti-pattern detection. */ + +import { PracticeGroup } from './analytics-types'; + +/** + * Where a rule was loaded from, in order of precedence (lowest to highest): + * - 'built-in': shipped with the extension (dist/rules/) + * - 'personal': user-level rules (~/.ai-engineer-coach/rules/) + * - 'project': workspace-level rules (.ai-engineer-coach/rules/ in the workspace root) + */ +export type RuleSource = 'built-in' | 'personal' | 'project'; + +/** + * Raw data emitted by a detector function. + * Detector functions ONLY compute metrics -- they never decide whether to flag. + * The rule engine evaluates thresholds from the .md rule file against this data. + */ +export interface DetectorEmission { + /** Number of flagged / matching items */ + count: number; + /** Total items in scope (requests or sessions) */ + total: number; + /** count / total (0-1), pre-computed for convenience */ + ratio: number; + /** Sample items for the examples template */ + examples: string[]; + /** Arbitrary extra data for template interpolation (e.g. {{extra.avgGap}}) */ + extra: Record; + /** When the detector computes dynamic severity, pass the actual severity here. + * If omitted the rule's static `severity:` field is used. */ + dynamicSeverity?: RuleSeverity; + /** Raw matched rows for building rich occurrence details */ + matchedRows?: Record[]; +} + +/** + * Severity level for a rule: determines scoring penalty weight. + * - high: 12-point penalty (critical issues) + * - medium: 7-point penalty (important issues) + * - low: 3-point penalty (minor issues) + */ +export type RuleSeverity = 'high' | 'medium' | 'low'; + +/** + * What data the rule operates on: requests (flat list) or sessions (grouped). + * Rules can also need both. + */ +export type RuleScope = 'requests' | 'sessions' | 'both'; + +/** + * A single condition in a rule's detection logic. + * Conditions are composed to build the full detection pipeline. + */ +export interface RuleCondition { + /** What to filter/check */ + type: + | 'filter-requests' // filter requests matching criteria + | 'filter-sessions' // filter sessions matching criteria + | 'ratio-check' // check a ratio (filtered / total) against a threshold + | 'count-check' // check a count against a minimum + | 'aggregate' // aggregate a value across requests/sessions + | 'group-by' // group by a field and check distribution + | 'regex-match' // match a regex pattern against request text + | 'field-check' // check a field value + | 'custom'; // custom logic (for complex detectors that can't be expressed declaratively) + + /** The field to operate on */ + field?: string; + + /** Comparison operator */ + op?: 'lt' | 'gt' | 'lte' | 'gte' | 'eq' | 'ne' | 'includes' | 'not-includes' | 'matches' | 'not-matches'; + + /** Value to compare against */ + value?: string | number | boolean; + + /** For regex-match conditions */ + pattern?: string; + flags?: string; + + /** For ratio-check: what constitutes the numerator */ + numerator?: string; + /** For ratio-check: the threshold to compare against */ + threshold?: number; + + /** For aggregate: the aggregation function */ + aggregation?: 'count' | 'sum' | 'avg' | 'max' | 'min' | 'ratio' | 'unique-count'; + + /** Minimum sample size before this condition applies */ + minSample?: number; + + /** Sub-conditions for complex logic */ + conditions?: RuleCondition[]; + + /** How to combine sub-conditions */ + logic?: 'and' | 'or'; +} + +/** + * Template variables available in rule description/suggestion strings. + * These are populated at runtime from the detection results. + */ +export interface RuleTemplateVars { + count: number; // number of occurrences (flagged items) + total: number; // total items checked + ratio: number; // count / total (0-1) + pct: string; // ratio formatted as percentage string + extra: Record; // additional rule-specific vars +} + +/** + * A fully parsed detection rule. This is the in-memory representation + * of a rule markdown file. + */ +export interface DetectionRule { + /** Unique rule identifier, e.g. 'lazy-prompting' */ + id: string; + + /** Human-readable name, e.g. 'Lazy Prompting' */ + name: string; + + /** Practice group this rule belongs to */ + group: PracticeGroup; + + /** Severity when triggered */ + severity: RuleSeverity; + + /** Whether this rule requires IDE-specific context (e.g. VS Code data) */ + requiresIdeContext: boolean; + + /** What data scope the rule needs */ + scope: RuleScope; + + /** Short description of what this rule detects */ + description: string; + + /** Template for the problem description shown when triggered. + * Supports {{count}}, {{total}}, {{pct}}, {{extra.xxx}} placeholders. */ + descriptionTemplate: string; + + /** Template for the suggestion shown when triggered */ + suggestionTemplate: string; + + /** Template for generating example strings */ + exampleTemplate: string; + + /** Maximum number of examples to show */ + maxExamples: number; + + /** The detection logic expressed as conditions */ + conditions: RuleCondition[]; + + /** Threshold constants used by this rule's conditions */ + thresholds: Record; + + /** Named regex pattern lists (e.g. patterns.frustration: ["!{3,}", "\\?{3,}"]) */ + patterns: Record; + + /** Named file extension groups (e.g. fileTypes.documentation: ["md", "txt", "rst"]) */ + fileTypes: Record; + + /** Parent rule ID for inheritance (extends: in frontmatter) */ + extendsRule?: string; + + /** Inline test fixtures */ + tests: Array<{ input: Record; expect: 'triggered' | 'clean' }>; + + /** Where this rule was loaded from (built-in, personal, or project) */ + source: RuleSource; + + /** Absolute path to the .md file this rule was loaded from (empty for in-memory rules) */ + sourceFilePath: string; + + /** Version of this rule definition */ + version: number; + + /** Tags for filtering/categorization */ + tags: string[]; + + /** The raw markdown source */ + rawSource: string; +} + +/** + * Result of evaluating a rule against data. + */ +export interface RuleEvalResult { + ruleId: string; + triggered: boolean; + occurrences: number; + total: number; + ratio: number; + severity: RuleSeverity; + description: string; + suggestion: string; + examples: string[]; + templateVars: RuleTemplateVars; +} + +/** + * Summary stats for a rule shown in the rule editor. + */ +export interface RulePreviewStats { + ruleId: string; + ruleName: string; + triggered: boolean; + occurrences: number; + total: number; + pct: number; + severity: RuleSeverity; + group: PracticeGroup; + previewDescription: string; + previewExamples: string[]; +} diff --git a/packages/ai-coach-analytics/src/types/session-types.ts b/packages/ai-coach-analytics/src/types/session-types.ts new file mode 100644 index 0000000..e982e02 --- /dev/null +++ b/packages/ai-coach-analytics/src/types/session-types.ts @@ -0,0 +1,182 @@ +/*--------------------------------------------------------------------------------------------- + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the MIT License. See LICENSE in the project root for license information. + *--------------------------------------------------------------------------------------------*/ + +export interface ToolConfirmation { + toolId: string; + confirmationType: number; + autoApproveScope?: string; + isTerminal: boolean; + commandLine?: string; +} + +export interface CompactionEvent { + mode: 'full' | 'simple'; + numRounds: number; + numRoundsSinceLastSummarization: number; + contextLengthBefore: number; + durationMs: number; + model: string; + outcome: string; +} + +/** Authoritative per-model usage totals reported at the session level + * (e.g. Copilot CLI emits these in `session.shutdown.modelMetrics`). + * Used for billing math when present; preferred over per-request data + * for sessions where per-request input is not available. */ +export interface ModelUsage { + /** Uncached input tokens billed at the input rate */ + inputTokens: number; + /** Output tokens billed at the output rate */ + outputTokens: number; + /** Cached tokens read (cheap rate) */ + cacheReadTokens: number; + /** Cached tokens written / cache creation (cache-write rate) */ + cacheWriteTokens: number; + /** Reasoning tokens (subset of output for some models, kept for diagnostics) */ + reasoningTokens?: number; +} + +export interface TodoItem { + id: number; + title: string; + status: 'not-started' | 'in-progress' | 'completed'; +} + +export interface SessionRequest { + requestId: string; + timestamp: number | null; + messageText: string; + responseText: string; + isCanceled: boolean; + agentName: string; + agentMode: string; + modelId: string; + toolsUsed: string[]; + editedFiles: string[]; + referencedFiles: string[]; + slashCommand: string; + variableKinds: Record; + customInstructions: string[]; + skillsUsed: string[]; + firstProgress: number | null; + totalElapsed: number | null; + messageLength: number; + responseLength: number; + userCode: CodeBlock[]; + aiCode: CodeBlock[]; + toolConfirmations: ToolConfirmation[]; + /** Prompt token count — from result.metadata.promptTokens (LAST agentic round's input only, + * not cumulative across all rounds). Cumulative input tokens are not persisted by VS Code. */ + promptTokens: number | null; + /** Completion token count — from request.completionTokens (CUMULATIVE across all agentic rounds) + * or metadata.outputTokens (last round only) as fallback for older sessions. */ + completionTokens: number | null; + /** Cached tokens read (subset of `promptTokens` when present); billed at cached rate */ + cacheReadTokens: number | null; + /** Cached tokens written / cache creation (subset of `promptTokens` when present); billed at cache-write rate */ + cacheWriteTokens: number | null; + /** Compaction/summarization event attached to this request, if one occurred */ + compaction: CompactionEvent | null; + /** Final TODO list state at the end of this request (from manage_todo_list tool calls) */ + todoSnapshot: TodoItem[] | null; + /** Precomputed work-type classification (feature, bug fix, refactor, etc.) */ + workType: string; + /** Reasoning / thinking effort for reasoning-capable models, when known. + * Sources: + * - Copilot CLI: `session.start.data.reasoningEffort` and `session.model_change.data.reasoningEffort` + * (carried forward to every turn under that effort setting). + * - Other harnesses: inferred from a known set of model name suffixes + * (e.g. `claude-opus-4.7-xhigh` → 'max', `claude-opus-4.7-high` → 'high'). + * Only set when the source actually exposes the value — never guessed. + * Values: 'max' | 'high' | 'medium' | 'low'. `null`/missing = unknown. */ + reasoningEffort?: 'max' | 'high' | 'medium' | 'low' | null; + /** Per-request finalization state — distinguishes requests that never had + * a chance to produce token data (still in-flight, or completed with an + * error like cancellation, network failure, length limit) and requests + * that finished successfully but where the source/harness simply did not + * record token usage (e.g. Xcode, or some 2026-04 VS Code chat requests + * for `copilot/auto` / `copilot/gpt-5.4`) — from genuine data gaps where + * the request finished, the harness normally would record tokens, but + * none are present. The first three are excluded from the coverage + * denominator; the last counts as `missing`. + * pending — request never finalized (no result object) + * errored — finalized with an error (canceled, network, length, etc.) + * no-data — finalized successfully but no token data was recorded by + * the harness (parser limitation that cannot be fixed + * from the source data) */ + endState?: 'pending' | 'errored' | 'no-data'; +} + +export interface CodeBlock { + language: string; + loc: number; +} + +/** Session-level disposition. Used to distinguish "missing token data + * because the session never closed" (active) and "missing because the + * user aborted before the model responded" (aborted) from genuine + * parser-coverage gaps (unknown). */ +export type SessionEndReason = 'shutdown' | 'active' | 'aborted' | 'unknown'; + +export interface Session { + sessionId: string; + workspaceId: string; + workspaceName: string; + location: string; + harness: string; + creationDate: number | null; + lastMessageDate: number | null; + requestCount: number; + requests: SessionRequest[]; + /** Authoritative per-model usage totals reported at session close (Copilot CLI). + * Preferred over per-request token data for billing math when present. */ + modelUsage?: Record; + /** How the session ended. Drives "pending" vs "missing" classification + * in the Token Coverage view: active/aborted sessions cannot have full + * token data and so are excluded from the missing% denominator. */ + endReason?: SessionEndReason; + /** True if the session shows runtime evidence of running inside a devcontainer + * (Codespaces / Remote-Containers). Detected by scanning request data for + * `/workspaces/...` paths in terminal commands, edited files, and referenced + * files. This is more reliable than checking the host disk for a + * `.devcontainer/` folder, because Codespaces never mount the project on + * the host. */ + hasDevcontainer?: boolean; + /** Size (in characters) of the workspace's `.github/copilot-instructions.md` + * file at parse time, when discoverable. `0` means the file is missing or + * empty; `undefined` means the workspace folder path could not be resolved + * (e.g. CLI session without `cwd`, or multi-root workspace). Used by the + * `instruction-bloat` rule to detect always-on-context bloat. */ + customInstructionsBytes?: number; + /** Resolved project root for CLI harnesses that record cwd separately from + * launch location/source. Used by config-health workspace scans. */ + workspaceRootPath?: string; + /** How the session was launched (Claude only, currently). + * interactive — user typed `claude` in a terminal or used Claude Desktop. + * programmatic — spawned by another tool via the SDK (e.g. GitHub Copilot + * agent mode), MCP, GitHub Action, etc. + * Stored for diagnostics; all Claude sessions use the `Claude` harness + * regardless of launcher kind. */ + launcherKind?: 'interactive' | 'programmatic'; + /** Raw `entrypoint` value from Claude JSONL (`cli`, `sdk-ts`, `sdk-py`, + * `mcp`, `claude-code-github-action`, etc.). Stored for diagnostics so the + * classification can be re-evaluated without re-parsing if the allow-list + * changes. `undefined` for non-Claude sessions and very old Claude versions + * that did not record this field. */ + entrypoint?: string; +} + +export interface Workspace { + id: string; + name: string; + path: string; +} + +export interface DateFilter { + fromDate?: string; + toDate?: string; + workspaceId?: string; + harness?: string; +} diff --git a/packages/ai-coach-analytics/tsconfig.json b/packages/ai-coach-analytics/tsconfig.json new file mode 100644 index 0000000..bdb9327 --- /dev/null +++ b/packages/ai-coach-analytics/tsconfig.json @@ -0,0 +1,8 @@ +{ + "extends": "../../tsconfig.base.json", + "compilerOptions": { + "lib": ["ES2022"], + "types": ["node"] + }, + "include": ["src"] +} diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 7dc70f3..f6ab47d 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -10,12 +10,15 @@ importers: apps/server: dependencies: + '@agents_fleet/ai-coach-analytics': + specifier: workspace:* + version: link:../../packages/ai-coach-analytics '@agents_fleet/shared': specifier: workspace:* version: link:../../packages/shared '@anthropic-ai/sdk': specifier: ^0.100.1 - version: 0.100.1 + version: 0.100.1(zod@4.4.3) '@homebridge/node-pty-prebuilt-multiarch': specifier: ^0.13.1 version: 0.13.1 @@ -127,6 +130,31 @@ importers: specifier: ^5.4.2 version: 5.4.21(@types/node@25.9.1) + packages/ai-coach-analytics: + dependencies: + leo-profanity: + specifier: ^1.9.0 + version: 1.9.0 + zod: + specifier: ^4.4.3 + version: 4.4.3 + devDependencies: + '@eslint/js': + specifier: ^9.9.1 + version: 9.39.4 + '@types/node': + specifier: ^22.13.5 + version: 22.19.19 + eslint: + specifier: ^9.9.1 + version: 9.39.4 + typescript: + specifier: ^5.6.3 + version: 5.9.3 + typescript-eslint: + specifier: ^8.4.0 + version: 8.60.0(eslint@9.39.4)(typescript@5.9.3) + packages/shared: devDependencies: '@eslint/js': @@ -1596,6 +1624,9 @@ packages: resolution: {integrity: sha512-buRG0fpBtRHSTCOASe6hD258tEubFoRLb4ZNA6NxMVHNw2gOcwHo9wyablzMzOA5z9xA9L1KNjk/Nt6MT9aYow==} engines: {node: '>= 0.6'} + french-badwords-list@1.0.7: + resolution: {integrity: sha512-H1ziKs2PJh2+UXZ9oCGJ/rRQpsI9NBykGf2Sc7WaKaj1OnWFuBXfsvANTdRcfVmOghGQaUmRyZ1hJOPbDpy04Q==} + fresh@0.5.2: resolution: {integrity: sha512-zJ2mQYM18rEFOudeV4GShTGIQ7RbzA7ozbU9I/XBpm7kqgMywgmylMwXHxZJmkVoYkna9d2pVXVXPdYTP9ej8Q==} engines: {node: '>= 0.6'} @@ -1752,6 +1783,10 @@ packages: keyv@4.5.4: resolution: {integrity: sha512-oxVHkHR/EJf2CNXnWxRLW6mg7JyCCUcG0DtEGmL2ctUo1PNTin1PUil+r/+4r5MpVgC/fn1kjsx7mjSujKqIpw==} + leo-profanity@1.9.0: + resolution: {integrity: sha512-vMrrrjsbT+fA5I/1rlBEVT5YjJsw1ASIVF8/xBEdZ6ylsg5AEIBrEvHxwe5XAmfVuTBi7aw2KcO2L3s9ddpKzw==} + engines: {node: '>=18'} + levn@0.4.1: resolution: {integrity: sha512-+bT2uH4E5LGE7h/n3evcS/sQlJXCpIp6ym8OWJ5eV6+67Dsql/LaaT7qJBAt2rzfoa/5QBGBhxDix1dMt2kQKQ==} engines: {node: '>= 0.8.0'} @@ -2042,6 +2077,10 @@ packages: engines: {node: '>=18.0.0', npm: '>=8.0.0'} hasBin: true + russian-bad-words@0.5.0: + resolution: {integrity: sha512-euNvEYki6iYYpkNbeudW+lEMMYGEmN7EBwVF8ezlbv0bZoQpVYB7W10cCeUIGV7Ed50sJynLQ0c559q5iI0ejQ==} + engines: {node: '>=10'} + safe-buffer@5.2.1: resolution: {integrity: sha512-rp3So07KcdmmKbGvgaNxQSJr7bGVSVk5S9Eq1F+ppbRo70+YeaDxkw5Dd8NPN+GD6bjnYm2VuPuCXmpuYvmCXQ==} @@ -2376,12 +2415,17 @@ packages: resolution: {integrity: sha512-rVksvsnNCdJ/ohGc6xgPwyN8eheCxsiLM8mxuE/t/mOVqJewPuO1miLpTHQiRgTKCLexL4MeAFVagts7HmNZ2Q==} engines: {node: '>=10'} + zod@4.4.3: + resolution: {integrity: sha512-ytENFjIJFl2UwYglde2jchW2Hwm4GJFLDiSXWdTrJQBIN9Fcyp7n4DhxJEiWNAJMV1/BqWfW/kkg71UDcHJyTQ==} + snapshots: - '@anthropic-ai/sdk@0.100.1': + '@anthropic-ai/sdk@0.100.1(zod@4.4.3)': dependencies: json-schema-to-ts: 3.1.1 standardwebhooks: 1.0.0 + optionalDependencies: + zod: 4.4.3 '@babel/code-frame@7.29.7': dependencies: @@ -3816,6 +3860,9 @@ snapshots: forwarded@0.2.0: {} + french-badwords-list@1.0.7: + optional: true + fresh@0.5.2: {} fs-constants@1.0.0: {} @@ -3945,6 +3992,11 @@ snapshots: dependencies: json-buffer: 3.0.1 + leo-profanity@1.9.0: + optionalDependencies: + french-badwords-list: 1.0.7 + russian-bad-words: 0.5.0 + levn@0.4.1: dependencies: prelude-ls: 1.2.1 @@ -4245,6 +4297,9 @@ snapshots: '@rollup/rollup-win32-x64-msvc': 4.60.4 fsevents: 2.3.3 + russian-bad-words@0.5.0: + optional: true + safe-buffer@5.2.1: {} safer-buffer@2.1.2: {} @@ -4596,3 +4651,5 @@ snapshots: yaml@1.10.3: {} yocto-queue@0.1.0: {} + + zod@4.4.3: {} diff --git a/screenshots/AI_Coach_Analytics_Anti_Patterns.png b/screenshots/AI_Coach_Analytics_Anti_Patterns.png new file mode 100644 index 0000000..4c2e6c5 Binary files /dev/null and b/screenshots/AI_Coach_Analytics_Anti_Patterns.png differ diff --git a/screenshots/AI_Coach_Analytics_Dashboard.png b/screenshots/AI_Coach_Analytics_Dashboard.png new file mode 100644 index 0000000..bfc8c8b Binary files /dev/null and b/screenshots/AI_Coach_Analytics_Dashboard.png differ diff --git a/screenshots/AI_Coach_Analytics_Detailed.png b/screenshots/AI_Coach_Analytics_Detailed.png new file mode 100644 index 0000000..f8d8863 Binary files /dev/null and b/screenshots/AI_Coach_Analytics_Detailed.png differ diff --git a/screenshots/AI_Coach_Analytics_Patterns.png b/screenshots/AI_Coach_Analytics_Patterns.png new file mode 100644 index 0000000..a828be0 Binary files /dev/null and b/screenshots/AI_Coach_Analytics_Patterns.png differ diff --git a/screenshots/AI_Coach_Analytics_SDLC.png b/screenshots/AI_Coach_Analytics_SDLC.png new file mode 100644 index 0000000..545ae55 Binary files /dev/null and b/screenshots/AI_Coach_Analytics_SDLC.png differ diff --git a/screenshots/AI_Coach_Analytics_Scorecard.png b/screenshots/AI_Coach_Analytics_Scorecard.png new file mode 100644 index 0000000..8e3108e Binary files /dev/null and b/screenshots/AI_Coach_Analytics_Scorecard.png differ diff --git a/screenshots/AI_Coach_Analytics_Timeline.png b/screenshots/AI_Coach_Analytics_Timeline.png new file mode 100644 index 0000000..f54df8b Binary files /dev/null and b/screenshots/AI_Coach_Analytics_Timeline.png differ diff --git a/screenshots/AI_Coach_Analytics_Timeline_Wide.png b/screenshots/AI_Coach_Analytics_Timeline_Wide.png new file mode 100644 index 0000000..760d7f4 Binary files /dev/null and b/screenshots/AI_Coach_Analytics_Timeline_Wide.png differ