Chat: Use Embeddings for Intent Classification, Not Dynamic Parameter Extraction

[goodtocode]

Use Embeddings for Intent Classification, Not Dynamic Parameter Extraction

Background

The Level 6 semantic classification work adds embedding-based matching between deterministic rule routing and the Microsoft Agent Framework fallback.

The intended routing hierarchy is:

User message
|
v
Deterministic rule classifier
|
+--> Known intent with valid captures --> Direct typed tool route
|
+--> Rule miss
|
v
Semantic intent classifier
|
+--> Non-parameterized intent --> Direct typed tool route
|
+--> Parameterized or complex request
|
v
MAF tool-calling layer
|
v
Typed application validation
|
+--> Valid arguments --> Tool execution
+--> Missing arguments --> Follow-up question
+--> Invalid arguments --> Correction or fallback

The current semantic classifier intentionally supports only non-parameterized intents. It should not be extended to generate dynamic tool parameters from embeddings.

This issue replaces the earlier proposal to use embeddings for parameter extraction.

Decision

Embeddings are used for semantic intent classification, not for dynamic parameter extraction.

This decision applies to both template-derived products:

• crucible-web
• agent-framework-quick-start

The products share the same architectural boundary:

• Deterministic parsing handles exact, structured, and known parameterized requests.
• Embeddings identify semantic intent for novel phrasing.
• MAF tool calling handles flexible argument composition.
• Application and domain code validate, authorize, and resolve all tool arguments.
• Chat follow-up messages collect missing or ambiguous values.

Why embeddings should not extract parameters

Embeddings preserve semantic similarity, not exact structured values.

For example:

select pipeline 123e4567-e89b-12d3-a456-426614174000

contains an exact GUID that must be preserved character-for-character. An embedding can identify that the message resembles pipeline selection, but it cannot reliably return the exact GUID.

The same limitation applies to:

• GUIDs.
• Dates.
• Numeric thresholds.
• URLs.
• Resource names.
• Search queries.
• Credentials or connection references.
• Multiple arguments.
• Negation and scope.
• Security-sensitive identifiers.

Two messages with entirely different identifiers may produce nearly identical embeddings because their semantic intent is the same. That is useful for intent matching but unsafe for parameter extraction.

Embeddings must not be treated as a source of exact values, typed arguments, authorization decisions, or security-sensitive data.

Recommended responsibility boundaries

Responsibility	Preferred mechanism
Recognize exact known phrasing	Deterministic rules
Extract obvious typed values	Deterministic capture parsing
Recognize novel intent	Embeddings
Select a tool for a novel parameterized request	MAF tool calling
Produce flexible natural-language arguments	MAF tool schema and constrained LLM output
Validate arguments	Application and domain code
Resolve fuzzy entity names	Optional semantic entity search plus authorization
Handle missing or ambiguous values	Chat follow-up
Enforce authorization	Application and database policy

Current behavior

The current rule classifier continues to handle parameterized requests such as:

• Select a pipeline by ID.
• Get an actor by ID.
• Get a pipeline by name.
• Search the web for a supplied query.
• Select a timeline by code.

Known structured requests continue to use deterministic captures and direct typed routing.

The semantic classifier handles non-parameterized intent recognition such as:

• List actors.
• List pipelines.
• Show recent messages.
• List playbooks.
• Show recent executions.

Parameterized or complex requests that do not match deterministic rules continue to fall through to MAF. MAF can select the appropriate tool and propose typed arguments using the registered tool schema.

Proposed implementation

Rule-first routing

Keep deterministic classification as the first routing tier.

When a rule match includes valid captures:

1. Return the classified intent.
2. Preserve the captured values.
3. Validate the values through the existing application path.
4. Route directly to the typed tool or command.

Semantic intent routing

When deterministic classification misses:

1. Generate an embedding for the user message.
2. Search the intent embedding store.
3. Apply the configured similarity threshold.
4. Resolve the matched intent against the current intent catalog.
5. Return a match only for intents that do not require captures.
6. Route the valid match directly.
7. Fall through to MAF when no valid semantic match exists.

MAF parameter handling

For parameterized or complex requests:

1. Provide MAF with the registered tool catalog and typed schemas.
2. Allow the model to select the appropriate tool.
3. Allow the model to propose structured arguments.
4. Validate those arguments in application code.
5. Apply tenant, owner, and authorization checks.
6. Ask a follow-up question when required values are missing or ambiguous.
7. Invoke the tool only after validation succeeds.

The model proposes a tool invocation. It does not become the authority for validity or authorization.

Semantic entity lookup

Embeddings may be useful after a parameter has been extracted.

For example:

show me the pipeline for customer onboarding

The system may extract:

pipelineName = customer onboarding

An optional semantic entity search can then find candidate pipeline records by name or description. That result must still pass:

• Tenant filtering.
• Owner filtering.
• Authorization.
• Exact or confidence-based candidate validation.
• Ambiguity handling.

This is semantic entity lookup, not semantic parameter extraction.

Entity lookup must never bypass application authorization or replace exact validation for sensitive identifiers.

Follow-up behavior

When a parameterized request is missing a required value, the chat cycle should ask a focused follow-up question.

Examples:

• Which pipeline would you like to select?
• What is the pipeline name?
• What web search query should I use?
• Which timeline code do you mean?

When multiple entities match:

• Ask the user to clarify.
• Do not select arbitrarily.
• Do not invoke the tool until the ambiguity is resolved.

When the next user message supplies the missing value:

• Preserve the pending intent context.
• Validate the supplied value.
• Complete the original route if validation succeeds.
• Return a normal tool result.

Requirements

Intent classification

• Keep deterministic rule classification as the first tier.
• Use embeddings only to identify semantic intent.
• Do not use embeddings to generate, reconstruct, or validate exact parameter values.
• Continue skipping parameterized intents in the semantic classifier unless a future design explicitly introduces a safe capture mechanism.
• Preserve the feature switch and disabled-by-default behavior.

MAF tool calling

• Use registered tool schemas for model-generated arguments.
• Keep tool descriptions and typed parameters authoritative for the model-facing contract.
• Do not introduce an independent JSON intent contract that can drift from the actual tools.
• Preserve existing governance and confirmation rules.
• Preserve the fallback from failed semantic matching to MAF.

Application validation

• Validate every model-proposed argument.
• Validate GUIDs, dates, numbers, enums, names, URLs, and other typed values.
• Apply tenant and owner filtering.
• Reject invalid or unauthorized values.
• Handle missing and ambiguous arguments through follow-up questions.
• Never pass unvalidated model output directly to a tool or database operation.

Observability

Capture structured telemetry for:

• Rule match.
• Semantic intent match.
• MAF tool-selection fallback.
• Parameter validation success or failure.
• Follow-up question.
• Ambiguous entity resolution.
• Tool invocation outcome.
• Semantic and MAF latency.
• Fallback reason.

Do not log secrets, credentials, tokens, private keys, or unnecessary sensitive message content.

Acceptance criteria

• Known parameterized requests continue to use deterministic capture parsing.
• Novel non-parameterized phrasing can use semantic intent classification.
• Novel parameterized phrasing can fall through to MAF tool calling.
• MAF-generated arguments are validated before tool execution.
• Invalid arguments never reach the target tool.
• Missing arguments produce a focused follow-up question.
• Ambiguous entity matches produce clarification rather than arbitrary routing.
• A follow-up message can complete a pending parameterized request.
• Semantic provider failure does not break chat routing.
• Rule matches remain faster than semantic or MAF processing.
• Tenant, owner, and authorization checks remain unchanged.
• The semantic feature remains disabled by default until evaluation approves enablement.
• No embedding is used as the authoritative representation of a dynamic parameter.
• Reqnroll scenarios cover deterministic capture, semantic non-parameterized routing, MAF parameter fallback, missing parameters, invalid parameters, and follow-up completion.
• Deterministic test generators and stores remain the default for automated tests.
• Live Azure validation remains a separate environment-level concern.

Suggested test scenarios

Deterministic parameter extraction

Given the user says a known parameterized phrase containing a valid identifier
When deterministic classification runs
Then the expected intent is returned
And the typed capture is present
And the direct tool route is invoked

Semantic non-parameterized classification

Given semantic classification is enabled
And the user uses novel phrasing for a non-parameterized intent
When semantic classification runs
Then the expected intent is returned
And no parameter extraction is required
And the direct tool route is invoked

Semantic parameterized request falls through to MAF

Given semantic classification is enabled
And the user uses novel phrasing for a parameterized intent
When deterministic classification does not match
Then semantic intent classification does not fabricate captures
And the request proceeds to MAF tool calling

MAF argument validation

Given MAF proposes typed tool arguments
When the application validates the arguments
Then valid arguments continue to the tool
And invalid arguments do not reach the tool

Missing parameter

Given the user expresses a parameterized intent without the required value
When tool routing evaluates the request
Then no tool is invoked
And the assistant asks only for the missing value

Ambiguous entity

Given the user supplies a name that matches multiple authorized entities
When entity resolution runs
Then no entity is selected arbitrarily
And the assistant asks the user to clarify

Follow-up completion

Given a pending parameterized intent exists
And the assistant asks for the missing value
When the user provides a valid value in the next message
Then the value is validated
And the original tool route is completed

Semantic provider failure

Given semantic classification is enabled
And embedding generation or search fails
When the classifier runs
Then the chat request does not crash
And the request follows the configured MAF fallback

Out of scope

• Using embeddings to extract exact dynamic parameters.
• Treating embedding similarity as authorization.
• Passing embedding results directly to tools.
• Replacing deterministic capture parsing.
• Replacing MAF tool calling with a separate parameter-generation protocol.
• Playbook evaluation metric embeddings.
• Azure AI Search migration.
• Automatic production enablement without Phase 8.3 evaluation.
• General conversational memory redesign.

Documentation requirements

• Update the shared product feature documentation in both repositories.
• State clearly that embeddings classify intent and do not extract dynamic parameters.
• Document the rule, semantic, MAF, validation, and follow-up boundaries.
• Keep durable architectural principles in governance documentation.
• Keep repository-specific implementation details and tactical test cases in product feature documentation.
• Update any outdated documentation that describes semantic embeddings as a parameter-extraction mechanism.

Definition of done

• This decision replaces the previous proposal to use embeddings for parameter extraction.
• Deterministic parameter routing remains covered by tests.
• Semantic non-parameterized routing remains covered by tests.
• MAF parameter fallback is covered by tests.
• Application validation is covered for invalid and unauthorized arguments.
• Missing and ambiguous values produce follow-up behavior.
• Provider failures fall back safely.
• Reqnroll scenarios pass in both repositories.
• Governance and product documentation reflect the agreed responsibility boundaries.
• Production enablement remains a separate Phase 8.3 decision.