Real-Time Cost Anomaly Detection with Budget Optimization Advisor

[github-actions]

Strategic Improvement

Proactive cost intelligence that detects spending anomalies (sudden spike, budget trending toward exhaustion) and suggests optimizations. Analyzes recent pipeline runs, identifies expensive stages/iterations, recommends model downshifts (Opus→Sonnet, Sonnet→Haiku) where quality is preserved. Alerts when budget utilization >80% with projected runway. Outputs actionable optimization plan.

Acceptance Criteria

• Monitors cost data in real-time, updates every pipeline completion
• Anomaly detection: cost spike >2σ from 7-day average, budget utilization >80%
• Optimization advisor: analyzes past runs, suggests cheaper model routing for low-complexity stages
• Alert mechanisms: stdout warning, file marker, webhook (optional)
• Dashboard integration: cost trends, anomaly markers, optimization suggestions
• Test suite validates anomaly detection and optimization recommendations

Context

• Priority: P3
• Complexity: fast
• Generated by: Strategic Intelligence Agent
• Strategy alignment: P3: Cost Efficiency

[github-actions]

⚓ Shipwright Pipeline Starting

Field	Value
Template	standard
Complexity	standard
Max Iterations	20
Triage	Cost anomaly detection extends existing sw-cost.sh with straightforward algorithms; clear scope, well-defined acceptance criteria, moderate dashboard and webhook integration complexity.
Runner	ubuntu-latest
Risk Level	unknown
Workflow	23428307013

The autonomous pipeline is processing this issue. Updates will be posted as stages complete.

[github-actions]

[github-actions]

📋 Implementation Plan

Click to expand full plan (654 lines)

Implementation Plan: Real-Time Cost Anomaly Detection with Budget Optimization Advisor

Goal Summary

Build a real-time cost anomaly detection system that:

• Monitors pipeline costs against 7-day baseline (detect spikes >2σ)
• Alerts when budget utilization exceeds 80%
• Recommends model routing optimizations (Opus→Sonnet→Haiku)
• Integrates with dashboard and daemon for automatic updates
• Provides actionable optimization plans with test validation

---

Design Reasoning

Alternatives Considered

Alternative 1: Pure Bash Implementation

• Pros: No new dependencies, single shell script
• Cons: Difficult unit testing, error-prone math, complex shell logic
• Rejected: Testing requirements mandate better testability

Alternative 2: Hybrid TypeScript + Bash (Recommended)

• Pros: Core logic testable with vitest, integrates naturally with bash pipeline via JSON, leverages existing test infrastructure
• Cons: Requires Node invocation from bash
• Chosen: Aligns with project conventions (vitest already configured), better maintainability

Alternative 3: Pure TypeScript/Node CLI

• Pros: Fully testable, modern tooling
• Cons: Requires wrapping all bash integrations, doesn't follow Shipwright's bash-native pattern
• Rejected: Over-engineers for bash-centric pipeline

Key Design Decisions

1. Statistical Foundation: Use mean + standard deviation (2σ threshold) for anomaly detection
2. Data Window: Analyze last 100 pipeline runs (not unlimited history) to bound memory usage
3. Optimization Scope: Focus on stage-level model routing (low-complexity stages can downshift)
4. Alert Channels: stdout + file marker (webhook optional, deferred to v2)
5. Dashboard Integration: New /api/cost-anomalies endpoint with WebSocket updates
6. Daemon Hook: Post-pipeline-completion callback to run detection

---

Files to Modify

New Files

• src/cost-anomaly-detector.ts — Core anomaly detection + optimization logic
• src/cost-anomaly-detector.test.ts — Comprehensive test suite (70+ tests)
• scripts/sw-cost-anomaly.sh — Bash wrapper and daemon integration
• src/types/cost-anomaly.ts — Shared TypeScript interfaces

✅ Task Checklist

• Task 1: Create src/types/cost-anomaly.ts with all TypeScript interfaces (CostDataPoint, AnomalyAlert, OptimizationSuggestion, CostAnalysisResult)
• Task 2: Implement statistical helper functions (mean, std dev, z-score) in src/cost-anomaly-detector.ts
• Task 3: Implement detectAnomalies() function with 2σ spike detection and 80% budget utilization alerts
• Task 4: Implement suggestOptimizations() function analyzing stage success rates for model downshift recommendations
• Task 5: Implement analyzeAndAlert() orchestrator function that reads cost/budget JSON and returns CostAnalysisResult
• Task 6: Create scripts/sw-cost-anomaly.sh bash wrapper that invokes Node module and outputs JSON
• Task 7: Integrate cost anomaly detection into scripts/sw-daemon.sh post-pipeline-completion hook
• Task 8: Add cost analysis config section to daemon-config.json defaults (anomaly threshold, budget warning threshold, history window)
• Task 9: Add /api/cost-anomalies endpoint to dashboard/server.ts
• Task 10: Add WebSocket event emission for real-time anomaly updates in dashboard
• Task 11: Create dashboard UI components for anomalies, suggestions, and budget runway visualization
• Task 12: Write 40-50 unit tests in src/cost-anomaly-detector.test.ts (statistical functions, anomaly detection, optimizations, edge cases)
• Task 13: Write 10-15 integration tests with realistic cost fixture data
• Task 14: Create scripts/sw-cost-anomaly-test.sh E2E test harness (3-5 scenarios: detection, dashboard, daemon integration)
• Task 15: Update .claude/CLAUDE.md with new shipwright cost analyze and shipwright cost anomalies commands
• Task 16: Add version constants and ensure consistency across all files (VERSION="2.0.0")
• Task 17: Run full test suite: npm test — ensure all 70+ tests pass, coverage targets met
• All 5 core functions implemented and exported from cost-anomaly-detector.ts
• Bash wrapper script functional and tested manually
• Dashboard endpoints responding with cost anomaly data

---

Generated by shipwright pipeline at 2026-03-23T08:42:00Z

[github-actions]

[github-actions]

📐 Architecture Decision Record

Click to expand ADR (924 lines)

Architecture Decision Record: Real-Time Cost Anomaly Detection with Budget Optimization Advisor

Context

Shipwright needs real-time visibility into pipeline cost anomalies and actionable optimization recommendations to prevent budget overruns and optimize model routing across agent teams.

Constraints:

• Daemon must remain reactive (post-pipeline hook, not blocking)
• Dashboard requires fast responses (cached data, not live computation)
• Testing must be comprehensive (70+ tests covering statistical edge cases)
• Must integrate seamlessly with existing cost tracking (~/.shipwright/costs.json, ~/.shipwright/budget.json)
• Must follow project conventions (Node.js, vitest, bash-native pipeline, CommonJS)

Problem:
Without anomaly detection, cost spikes go unnoticed until budget exhaustion. Without optimization recommendations, model routing remains manual. Dashboard cannot display real-time cost health.

---

Decision

Adopt a hybrid TypeScript + Bash architecture:

1. Core analysis logic: TypeScript module (src/cost-anomaly-detector.ts) with pure statistical functions (mean, stddev, z-score) and three orchestrated analyzers (anomaly detection, optimization advisor, result composition)
2. Bash integration layer: Single wrapper script (scripts/sw-cost-anomaly.sh) invokes Node module, handles file I/O, emits events
3. Daemon integration: Post-pipeline hook in sw-daemon.sh calls wrapper asynchronously (non-blocking)
4. Dashboard integration: Cached result files + /api/cost-anomalies endpoint + WebSocket broadcasts
5. Statistical approach: 2σ threshold for spike detection, 100-run rolling window (bounded memory), 95%+ success rate threshold for optimization recommendations

Why this approach:

• TypeScript core enables comprehensive unit testing (vitest), type safety, and easy debugging of statistical functions
• Bash wrapper keeps daemon-level logic simple and aligns with Shipwright's bash-native pipeline
• Cached results prevent dashboard API timeouts under concurrent pipeline load
• Async hook ensures anomaly detection never blocks pipeline completion
• Statistical approach avoids heuristic brittleness (data-driven, mathematically sound)

---

Alternatives Considered

1. Pure Bash Implementation

Approach: Single sw-cost-anomaly.sh script doing all analysis with awk and bc

Pros:

• No new runtime dependency
• Direct integration (no subprocess overhead)
• Single file to maintain

Cons:

• Bash math is fragile and error-prone (floating-point precision, quoting issues)
• No unit testing framework — cannot achieve 70+ test coverage
• Edge cases (variance=0, negative values) are hard to handle in shell
• Debugging statistical bugs in bash is painful

Trade-off Analysis:

• Simplicity: 6/10 (shell math is complex)
• Testability: 2/10 ← Violates acceptance criteria
• Maintainability: 4/10 (shell bugs are silent)

Rejected: Fails testing requirement explicitly stated in plan ("comprehensive validation of detection + recommendations").

---

Generated by shipwright pipeline design stage at 2026-03-23T08:44:55Z

[github-actions]

[github-actions]

🔨 Build started — shipwright loop with 20 max iterations, 1 agent(s), model: haiku

[github-actions]

[github-actions]

[github-actions]

[github-actions]