Chat: Level 6 Semantic Classification & Embedding

[goodtocode]

Level 6 / Phase 8+ — Semantic Classification & Embedding Store (draft, to discuss)

Goal: Extend IIntentClassifier with a SemanticIntentClassifier (or HybridIntentClassifier combining rule + semantic) so novel phrasings not in IntentDefinition.Examples can still match confidently, without hand-written regex growth.

Open design questions for discussion (not decided):

Embedding store: SQL Server as the baby step (native VECTOR type if your SQL Server/Azure SQL version supports it, else a plain table + brute-force cosine similarity) — behind an IIntentEmbeddingStore abstraction so Azure AI Search / Cosmos DB vCore can replace it later without touching the classifier contract. Need to confirm your target SQL Server/Azure SQL edition before assuming native vector support.
Embedding generation: which model/service generates the vectors (Azure OpenAI embeddings? something already in your MAF/MEAI stack?) and where that call lives (must stay out of Core.Application per the DDD boundary rule — likely Infrastructure.AgentFramework or a new Infrastructure.Embeddings).
Confidence threshold & fallback policy: what similarity score triggers deterministic routing vs. falling through to the MAF agent — needs a tuning methodology, not a guess.
Hybrid ordering: does rule classifier always run first (cheap, exact) with semantic only as a second pass on rule-miss, or do both run and highest-confidence wins? I'd lean toward rule-first-then-semantic-fallback for cost/latency, but open to your view.
Test harness: classifier accuracy likely needs its own offline eval harness (labeled exemplar set + accuracy score) separate from the live-AI E2E suite, since embedding quality isn't really an "E2E against MAF" concern.
I'm not proposing an implementation shape yet — just the shape of the discussion once Phase 7 lands.

Level 6 Semantic Classification: Executive Summary & Discussion Frame

Date: 2026-09-05
Status: Design Complete — Ready for Stakeholder Review
Phase Target: Phase 8+ (post-Phase 7 release)
Effort Estimate: ~2.5 developer-weeks (~100 hours)

---

The Problem

Currently, both projects use rule-based intent classification (Level 4):

User Prompt → Exact keyword/regex match → Intent Found or Not Found → MAF Agent Fallback

Limitation: Novel phrasings that don't match hardcoded examples fall through to the expensive/slow MAF agent tool-calling pipeline.

Example:

• Registered example: "create a new playbook"
• User says: "help me start a fresh cloud evaluation workflow"
• Result: No match → MAF agent called (4–5 sec latency, higher cost)

---

The Solution

Extend classification with semantic matching (Level 6):

User Prompt
  ↓
[Tier 1: Rule Classifier] ← Exact match (0ms, 100% confidence)
  ↓ [miss]
[Tier 2: Semantic Classifier] ← Vector similarity (200ms, probabilistic)
  ↓ [miss]
[Tier 3: MAF Agent] ← Full inference (4–5s, most expensive)

Benefits:

• ✅ Handle novel phrasings without regex growth
• ✅ Keep deterministic fast path (rules first)
• ✅ Moderate latency (200ms vs 4–5s fallback)
• ✅ Lower cost (single embedding query vs. full LLM tool-call)
• ✅ Future migration path (Azure AI Search, Cosmos DB vCore)

---

Key Design Decisions

1. Storage: SQL Server VECTOR (or Float Array Fallback)

• Why: One less service; already running SQL. Native VECTOR type available in SQL Server 2025+.
• Fallback: If using SQL Server 2022/2019, store as serialized float array; compute similarity in C#.
• Scale: Brute-force cosine similarity acceptable for ~2500 vectors (one scan is <100ms).

2. Embeddings Model: Azure OpenAI text-embedding-3-small

• Why: Already available in stack; mature; good for intent classification.
• Dimension: 1536-dim vectors.
• Cost: ~$0.02 per 1M tokens; seeding ~2500 embeddings ≈ $0.05–0.10 per app start/test run.

3. Source of Truth: IntentDefinition.Examples (Not Embeddings)

• Why: Embeddings are indexed cache, like SQL indexes. Examples are canonical.
• Benefit: Changing embedding model (3-small → 4) doesn't require code changes; just regenerate cache.
• No embeddings in source code: Generated dynamically at seeding/startup.

4. Hybrid Order: Rule → Semantic → MAF

• Why: Cost/latency pyramid: exact (free) → fuzzy (moderate) → agent (expensive).
• No both-running: Sequential fallback preserves determinism and cost efficiency.

5. Seeding Strategy: Two Approaches

Project	Approach	When	Use Case
crucible-web	Seed.StateOfYour integration	During business acceptance test setup	Embeddings regenerated per test; always in sync
agent-framework-quick-start	IHostedService on startup	On app start/restart	Idempotent; works in any deployment

6. Feature Flag: EnableSemantic = false by default

• Rollout safety: Deploy infrastructure dormant; enable after offline accuracy tuning.
• Fallback: If embeddings fail, app still works (rule-only classifier).

7. Confidence Threshold: 0.75 (Configurable)

• Rationale: Copilot analysis suggests 0.75 balances accuracy vs. false positives.
• Tuning: Offline evaluation harness with 200+ labeled examples determines optimal per use case.
• No hardcoding: Threshold in appsettings.json; adjust without code changes.

---

Open Questions for Design Review

Q1: Confidence Threshold

Proposal: Accept 0.75 as starting point; tune via offline evaluation in Phase 8.3.

• Offline eval: Run 200+ labeled exemplars through classifier
• Measure: Top-1 accuracy, false positives, mismatches
• Adjust threshold until accuracy > 85% and false positives < 2%
• Update appsettings.json before enabling flag

Your Input: Acceptable? Different baseline preferred?

---

Q2: Weighted Scoring

Proposal: Different weights for different embedding sources:

• Examples: 1.0 (users write these; highest trust)
• Descriptions: 0.8 (tool authors write these; less user-centric)
• Tool metadata: 0.6 (derived fields; lowest priority)

When: Implement in Phase 8.2 (classifier design already supports Weight field).

Your Input: Adopt this? Different weighting scheme?

---

Q3: Should IntentDefinition Include Descriptions?

Proposal: Add optional Description property (e.g., "Creates a playbook for SQL Server evaluation").

Benefit: Descriptions often capture intent semantically; embedding them increases matches.

Example:

new IntentDefinition(
    "CreatePlaybook",
    Examples: ["create a new playbook", "add playbook"],
    Description: "Creates a playbook for SQL Server or similar infrastructure evaluation",
    ...
);

When: Defer to Phase 8.3 (nice-to-have, not MVP).

Your Input: Include? Defer?

---

Q4: IIntentClassifier Async Contract

Current: IntentMatch? Classify(string message, ...)

Problem: Semantic matching requires async (embedding generation, DB query).

Option A: Extend to Async (Cleaner, but breaks downstream)

Task<IntentMatch?> ClassifyAsync(string message, ...);

• Routing service also goes async
• Interface-wide consistency
• ~4 hours refactor in each project

Option B: Decorator Pattern (Preserve sync, layer semantic)

// Keep IIntentClassifier sync
// SemanticClassifier is separate, called from routing service explicitly
// Routing service handles async

• No interface change
• More code in routing layer
• Slightly more complex flow

Recommendation: Option A (interface async) for architectural cleanliness.

Your Input: Accept Option A? Prefer Option B?

---

Q5: PlaybookEvaluationStep Embeddings

Mention in Brief: Design document mentions playbook evaluation steps may also benefit from embeddings (e.g., matching user metrics against expected evaluation criteria).

Proposal: Defer to Phase 9+ (separate epic). Focus Phase 8 solely on intent classification.

Your Input: Acceptable scope boundary? Need playbook embeddings sooner?

---

Q6: Timeline to Migrate to Azure AI Search

Abstraction Layer: IIntentEmbeddingStore abstracts the backend; SQL, Azure AI Search, Cosmos DB vCore all pluggable.

Migration Path: When scale exceeds ~50K embeddings, migrate to Azure AI Search.

• Estimated effort: 1–2 weeks (new IIntentEmbeddingStore impl, same routing logic)
• No impact on SemanticIntentClassifier or routing

Proposal: Document path now; execute in Phase 9+ when needed.

Your Input: Acceptable? Want earlier planning?

---

Implementation Roadmap

Phase 8.1: Foundation (Weeks 1–2)

• Abstractions: IEmbedding, IEmbeddingGenerator, IIntentEmbeddingStore
• SQL infrastructure: Entity, configuration, migration
• Azure OpenAI generator implementation
• DI registration + feature flag
• Deliverable: Infrastructure ready; no semantic routing yet (EnableSemantic: false)
• Tests: Unit tests for generator, store, similarity computation
• Cost: ~28 hours

Phase 8.2: Classifiers + Seeding (Weeks 3–4)

• SemanticIntentClassifier implementation
• HybridIntentClassifier wrapper with async rule-first classification
• crucible-web: Integrate into Seed.StateOfYourSqlServerRuntimeScenario
• agent-framework-quick-start: IHostedService for startup initialization
• Integration tests (Reqnroll) in both projects
• Deliverable: Functional semantic classification; not yet enabled in production
• Tests: Classifier unit + integration tests; seeding validation
• Cost: ~36 hours

Phase 8.3: Evaluation & Threshold Tuning (Weeks 5+)

• Offline accuracy evaluation harness (200+ labeled exemplars)
• Measure accuracy, false positives, latency
• Determine optimal confidence threshold
• E2E regression test suite
• Enable IntentClassification.EnableSemantic: true in config
• Monitor production; log semantic matches
• Deliverable: Semantic classification live; tuned to project-specific accuracy targets
• Tests: Full E2E suite; performance regression; accuracy benchmarks
• Cost: ~36 hours

Total: ~100 hours (~2.5 developer-weeks)

---

Risks & Mitigation

Risk	Mitigation
Azure OpenAI latency	Embedding requests are batched; ~10–20 per test/startup. Acceptable cost.
Confidence threshold too low	Offline evaluation calibrates before enablement. Monitor logged decisions post-launch.
SQL query performance	Brute-force similarity scan acceptable for ~2500 vectors. Add index if scale grows.
False positives	Threshold tuning in Phase 8.3; feature flag allows instant rollback.
Database migration	EF Core handles schema generation; tested in integration suite.
Two different implementations	Copy code with cross-project comments; refactor to shared NuGet in Phase 9.

---

Success Criteria

Phase 8.1 ✅

• Abstractions compile and are testable
• SQL migration runs without errors
• Azure OpenAI generator produces valid vectors

Phase 8.2 ✅

• Semantic classifier matches known phrasings with > 0.85 similarity
• Hybrid classifier falls through rule → semantic → MAF correctly
• Seeding (crucible) and startup (quick-start) both functional
• Reqnroll tests pass in both projects

Phase 8.3 ✅

• Offline eval harness measures > 85% top-1 accuracy
• False positive rate < 2%
• Feature flag enables semantic matching safely
• E2E chat tests pass with semantic matching enabled
• Latency acceptable (rule: <1ms, semantic: 150–300ms, fallback: 3–5s)

---

Questions for Stakeholders

1. Approve design direction? (SQL-backed embeddings, Azure OpenAI, hybrid classifier)
2. Accept proposed timeline? (~2.5 weeks, phased rollout)
3. Guidance on open questions 1–6 above?
4. Any additional use cases to consider (e.g., playbook evaluation, description embeddings)?
5. Approval to proceed with Phase 8.1 implementation?

---

Deliverables for This Design Phase

✅ Main Design Document (600+ lines)

• Full architecture, abstractions, implementations
• SQL schema, DI configuration, error handling
• Testing strategy, phase breakdown

✅ Quick Reference Guide (300+ lines)

• One-page architecture, configuration, checklist
• Cost estimates, observability, rollback strategy

✅ Implementation Comparison (350+ lines)

• Shared vs. project-specific code
• Seeding patterns (crucible vs. quick-start)
• File sync strategy, testing approach

✅ This Executive Summary

• Problem/solution framing
• Key decisions + rationale
• Open questions for discussion
• Roadmap + success criteria

---

Next Steps

1. Review: Stakeholders review this summary + linked design documents
2. Discuss: Address open questions 1–6 in team meeting
3. Decide: Approval to proceed with Phase 8.1
4. Create Tickets: Break Phase 8.1 into implementable work items
5. Kick Off: Begin coding Phase 8.1 (foundation)

---

Contact & Questions

Design Owner: [Your Team]
Documents:

• level-6-semantic-classification-design.md
• level-6-quick-reference.md
• level-6-implementation-comparison.md

Reference Conversation: [Copilot Design Conversation on Embeddings + Intent Classification]

---

Last Updated: 2026-09-05
Status: Ready for Stakeholder Review & Approval