Skip to content

Extend key entities metric to audio-native models #180

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
weiz9 wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Extend key entities metric to audio-native models #180

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
166 changes: 166 additions & 0 deletions configs/prompts/judge.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1071,3 +1071,169 @@ judge:
"summary": "<1-2 sentence summary for this turn>"
}}
]

audio_native_prompt: |
You are an expert evaluator analyzing whether an AI voice agent correctly **understood** the key entities a user provided during a spoken conversation.

This is a speech-to-speech / audio-native agent: there is no reliable speech-to-text transcript of the user to inspect. Instead, the evidence of what the agent actually understood is what it did with the information — primarily the **arguments it passed to its tool calls**, and secondarily any values it read back to the user. When a user says "my id is EM123" and the agent then calls a tool with "EMP124", the agent misheard the entity.

Your task:
1. For EACH user turn, identify all key entities in what the USER SAID (the "User said" text).
2. For each entity, find where the agent used it — a tool call argument (see the Conversation below) or a value the agent read back — and check whether the agent captured it CORRECTLY.
3. Mark each entity as correct or incorrect.
4. If the agent never used a given entity anywhere (no tool call argument and no read-back reference to it), there is no evidence to judge understanding — include the entity but mark it as skipped.
5. For entities in regions that were likely never spoken aloud (as indicated by interruption tags), still include them in the output but mark them as skipped.

## What Counts as an Entity
An entity must have a **specific, concrete value** — something that could be passed as an input to a program or tool (not an AI, but a script or database lookup). Ask yourself: could this value be stored in a variable and used programmatically?

- Names (people, places, organizations): e.g. "John Smith", "Austin", "Delta Airlines"
- Specific dates and times: e.g. "December 15th", "3:45 PM" — NOT vague references like "tomorrow morning" or "later today"
- Confirmation codes / reference numbers: e.g. "ABC123", "ZK3FFW"
- Flight numbers: e.g. "UA 204"
- Amounts and prices: use the specific value only, e.g. "$120" — for qualifier phrases like "under $120", only use the specific value
- Addresses: e.g. "123 Main Street"
- Phone numbers: e.g. "555-867-5309"
- Email addresses: e.g. "john@example.com"
- Other specific identifiers: seat numbers, loyalty numbers, booking IDs, employee IDs, etc.

**Not an entity:** vague temporal words ("tomorrow", "next week", "morning"), general descriptors ("the cheap flight", "a long trip"), or open-ended qualifiers ("less than an hour", "around noon").

## IMPORTANT: What This Metric Measures
This measures whether the agent **understood** the entity the user provided, as evidenced by how it used it. It is NOT a faithfulness or tool-correctness metric. Do NOT penalize:
- The agent choosing the wrong tool, or omitting a tool call it should have made
- The agent using an entity that came from a tool RESPONSE rather than from the user (only user-provided entities are evaluated here)
- Correct values that simply never got reused (mark those skipped, not incorrect)

Only mark an entity incorrect when the agent demonstrably used a **different value** than the user provided (e.g. user "EM123" → tool call "EMP124", user "March 25th" → tool call "2026-03-26").

## Understanding Tags in the User Text
The user text may contain non-spoken tags and markers. These are metadata — they were never said aloud and must not be treated as entities or evaluated.

### Audio-Direction Tags
Tags like [slow], [firm], [annoyed] describe how the words were meant to be spoken. Ignore them entirely.

### Interruption Tags
{interruption_tags_reference}

These markers indicate that parts of the user text may never have been spoken aloud, because the user was interrupted or talked over. An entity that was never spoken cannot have been understood, so you must NOT penalize for entities in regions that were likely not said — mark them as skipped.

**Key principle:** Only evaluate entities that were reasonably expected to have been spoken aloud AND that the agent had an occasion to use. If a tag indicates the user was interrupted before or during an entity, still include the entity but set `skipped: true` and explain why.

## User Turns to Evaluate
{user_turns}

## Conversation (with agent turns and tool calls/responses)
Use this to find where each user-provided entity was used by the agent. Tool call arguments are the primary evidence of what the agent understood.
{conversation}

## Correctness Criteria
- Entity must have been used correctly by the agent — the value passed to the tool call (or read back) must match what the user said.
- Numbers: "150" and "one hundred fifty" are equivalent
- Dates: "December 15th" and "Dec 15" are equivalent
- Names: Case-insensitive match. Phonetically equivalent spelling variants are acceptable on the **first mention** of a name (e.g., German Meyer/Meier/Maier/Mayer — all sound the same when spoken). Once the user has explicitly disambiguated the spelling (e.g., spelled it out letter by letter, corrected a prior spelling, or confirmed a specific spelling after the assistant read it back), subsequent usage must match that clarified spelling exactly. Flag if:
- The value the agent used clearly differs (different sound, not just different spelling)
- The agent used a different spelling variant **after** the user has explicitly clarified the correct spelling
- Non-Latin scripts: A name or identifier rendered in its native writing system (e.g. Kanji, Hangul, Arabic) is equivalent to its romanized/phonetic form if they represent the same spoken entity. For example, "山田" and "Yamada" should be treated as matching.
- Phonetic letter spellings: In languages that do not use the Latin alphabet, Roman/Latin letters are spoken using language-specific phonetic equivalents. These are correct, not errors (e.g., in Korean "에이" for "A" or "비" for "B").
- Number surface variants: Many languages have systematic spoken-form differences that do not change the underlying value and must not be penalized:
- Reversed digit order in spoken form (e.g., German "vierundzwanzig" = four-and-twenty = 24)
- Grammatical gender/agreement variants (e.g., Spanish "veintiún" vs "veintiuna" for 21)
- Large-unit grouping differences (e.g., Japanese 万-system: "ichi-man go-sen" / 一万五千 = 15,000)
- Honorific/title variants: "Ms." vs "Miss", "Mr." vs "Mister", and language equivalents (e.g., French "Madame" vs "Mme", Japanese "様" vs "-san") are acceptable variants of the same title.

Important note: The user text will often feature things formatted like "one two three" instead of "123". Your goal is to evaluate semantic equivalence — these are considered equivalent if the value matches what the agent used.

## Examples

**Example Input:**
User Turns to Evaluate:
Turn 1:
User said: `My employee id is E M one two three and my name is John Smith.`

Turn 2:
User said: `Book it for December 15th.`

Turn 3:
User said: `[slow] The code is X X F six O H, with the letter O, [assistant interrupts] not zero.`

Conversation:
Turn 1:
user: My employee id is E M one two three and my name is John Smith.
tool_call: lookup_employee({{'employee_id': 'EMP124'}})
Turn 2:
user: Book it for December 15th.
tool_call: create_booking({{'date': '2026-12-15'}})

**Example Response:**
[
{{
"turn_id": 1,
"entities": [
{{
"type": "employee_id",
"value": "E M one two three",
"captured_value": "EMP124",
"analysis": "User said EM123 but the agent called lookup_employee with EMP124 — misheard.",
"correct": false,
"skipped": false
}},
{{
"type": "name",
"value": "John Smith",
"captured_value": "not used",
"analysis": "The name was never passed to a tool call or read back, so there is no evidence of how the agent understood it.",
"correct": false,
"skipped": true
}}
],
"summary": "1 entity used (employee_id, incorrect). Name never used — skipped."
}},
{{
"turn_id": 2,
"entities": [
{{
"type": "date",
"value": "December 15th",
"captured_value": "2026-12-15",
"analysis": "Agent passed 2026-12-15 to create_booking — matches December 15th.",
"correct": true,
"skipped": false
}}
],
"summary": "1 entity correct."
}},
{{
"turn_id": 3,
"entities": [
{{
"type": "confirmation_code",
"value": "X X F six O H",
"captured_value": "not used",
"analysis": "Code appears before the [assistant interrupts] tag, but the agent never used it in a tool call — no evidence to judge, skipped.",
"correct": false,
"skipped": true
}}
],
"summary": "1 entity found; never used by the agent — skipped."
}}
]

## Response Format
Respond with a JSON object. Each turn entry must include the turn_id matching the turn number shown in the User Turns to Evaluate section above:
[
{{
"turn_id": <int: the turn number from the User Turns to Evaluate section>,
"entities": [
{{
"type": "<name|date|time|confirmation_code|flight_number|amount|address|phone|email|employee_id|etc...>",
"value": "<entity value as the user said it>",
"captured_value": "<the value the agent used in a tool call / read-back, or 'not used'>",
"analysis": "<brief reason; if skipped, explain why (never used by the agent, or in an interrupted region)>",
"correct": <true|false>,
"skipped": <true|false>
}}
],
"summary": "<1-2 sentence summary for this turn>"
}}
]
2 changes: 1 addition & 1 deletion src/eva/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,4 @@

# Bump metrics_version when changes affect metric computation (metrics code,
# judge prompts, pricing tables, postprocessor).
metrics_version = "2.2.0"
metrics_version = "2.3.0"
81 changes: 66 additions & 15 deletions src/eva/metrics/diagnostic/transcription_accuracy_key_entities.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,16 @@
"""Transcription accuracy key entities metric using LLM-as-judge (entire conversation).
"""Key entity accuracy metric using LLM-as-judge (entire conversation).

Measures whether the user's key entities (names, dates, confirmation codes,
amounts, etc.) survived intact through the agent's pipeline. The comparison
target depends on the pipeline type:

- **CASCADE**: compares what the user was instructed to say (``intended_user_turns``)
against what the agent's STT transcribed (``transcribed_user_turns``) — i.e.
transcription accuracy.
- **Audio-native (S2S / AUDIO_LLM)**: there is no reliable STT output to inspect,
so the agent's **tool-call arguments** (from ``conversation_trace``) become the
evidence of what it understood. If the user says "my id is EM123" and the agent
then calls a tool with "EMP124", the agent misheard the entity.

Debug metric for diagnosing model performance issues, not directly used in
final evaluation scores.
Expand All @@ -10,23 +22,29 @@
from eva.metrics.registry import register_metric
from eva.metrics.utils import (
aggregate_per_turn_scores,
format_transcript_with_tools,
make_rate_sub_metric,
parse_judge_response_list,
resolve_turn_id,
)
from eva.models.config import PipelineType
from eva.models.results import MetricScore


@register_metric
class TranscriptionAccuracyKeyEntitiesMetric(TextJudgeMetric):
"""LLM-based transcription accuracy metric for key entities only (entire conversation).
"""LLM-based key entity accuracy metric (entire conversation).

Evaluates STT transcription accuracy by comparing key entities (names, dates,
confirmation codes, amounts, etc.) between what the user was supposed to say
(intended_user_turns) and what STT transcribed (transcribed_user_turns).
Evaluates whether the user's key entities (names, dates, confirmation codes,
amounts, etc.) survived intact through the agent's pipeline. The comparison
target depends on the pipeline type:

Computes the ratio of correctly transcribed entities per turn:
- **CASCADE**: compares ``intended_user_turns`` against ``transcribed_user_turns``
(STT output) — transcription accuracy.
- **Audio-native (S2S / AUDIO_LLM)**: no reliable STT output exists, so the agent's
tool-call arguments (from ``conversation_trace``) are used as the evidence of
what it understood.

Computes the ratio of correctly captured entities per turn:
score = correct_entities / total_entities

Entity types evaluated:
Expand All @@ -48,11 +66,12 @@ class TranscriptionAccuracyKeyEntitiesMetric(TextJudgeMetric):
"""

name = "transcription_accuracy_key_entities"
version = "v0.3"
description = "Debug metric: LLM judge evaluation of STT key entity transcription accuracy for entire conversation"
version = "v0.4"
description = (
"Debug metric: LLM judge evaluation of user key entity accuracy (STT in cascade, tool calls in audio-native)"
)
category = "diagnostic"
exclude_from_pass_at_k = True
supported_pipeline_types = frozenset({PipelineType.CASCADE})
rating_scale = None # Custom scoring (not 1-3 scale)
default_aggregation = "mean"

Expand All @@ -70,9 +89,7 @@ async def compute(self, context: MetricContext) -> MetricScore:
MetricScore with aggregated score and per-turn details
"""
try:
turns_to_evaluate = self._get_turns_to_evaluate(context)
user_turns_text = self._format_user_turns(turns_to_evaluate, context)
prompt = self.get_judge_prompt(user_turns=user_turns_text)
prompt, turns_to_evaluate = self._build_prompt_and_turns(context)

response_text = await self._call_judge_raw(prompt, context)
turn_evaluations = parse_judge_response_list(response_text)
Expand Down Expand Up @@ -206,21 +223,55 @@ def _build_per_entity_type_sub_metrics(
)
return sub_metrics

def _build_prompt_and_turns(self, context: MetricContext) -> tuple[str, list[int]]:
"""Select the comparison source and judge prompt based on pipeline type.

CASCADE compares intended vs STT-transcribed user turns. Audio-native
pipelines (S2S / AUDIO_LLM) have no reliable STT, so the user's intended
turns are compared against the agent's tool-call arguments (from
``conversation_trace``), which are the evidence of what it understood.

Returns:
Tuple of (rendered judge prompt, sorted turn IDs to evaluate).
"""
if context.is_audio_native:
turn_ids = self._get_audio_native_turns(context)
prompt = self.get_judge_prompt(
prompt_key="audio_native_prompt",
user_turns=self._format_intended_user_turns(turn_ids, context),
conversation=format_transcript_with_tools(context.conversation_trace),
)
return prompt, turn_ids

turn_ids = self._get_turns_to_evaluate(context)
prompt = self.get_judge_prompt(user_turns=self._format_user_turns(turn_ids, context))
return prompt, turn_ids

@staticmethod
def _get_turns_to_evaluate(context: MetricContext) -> list[int]:
"""Return sorted turn IDs present in both tts_text_user and transcript_user."""
"""Return sorted turn IDs present in both intended and transcribed user turns."""
return sorted(context.intended_user_turns.keys() & context.transcribed_user_turns.keys())

@staticmethod
def _get_audio_native_turns(context: MetricContext) -> list[int]:
"""Return sorted intended user turn IDs (ground truth for audio-native pipelines)."""
return sorted(context.intended_user_turns.keys())

@staticmethod
def _format_user_turns(turn_ids: list[int], context: MetricContext) -> str:
"""Format user turns for the prompt."""
"""Format expected-vs-transcribed user turn pairs for the cascade prompt."""
return "\n\n".join(
f"Turn {tid}:\n"
f'Expected: "{context.intended_user_turns[tid]}"\n'
f'Transcribed: "{context.transcribed_user_turns[tid]}"'
for tid in turn_ids
)

@staticmethod
def _format_intended_user_turns(turn_ids: list[int], context: MetricContext) -> str:
"""Format the user's intended utterances for the audio-native prompt."""
return "\n\n".join(f'Turn {tid}:\nUser said: "{context.intended_user_turns[tid]}"' for tid in turn_ids)

async def _call_judge_raw(self, prompt: str, context: MetricContext) -> str | None:
"""Call LLM judge and return raw response text.

Expand Down
4 changes: 2 additions & 2 deletions tests/fixtures/metric_signatures.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,8 +86,8 @@
"TranscriptionAccuracyKeyEntitiesMetric": {
"name": "transcription_accuracy_key_entities",
"prompt_hash": "877dfa61f232",
"source_hash": "70e92b2f7221",
"version": "v0.3"
"source_hash": "1be843cc964d",
"version": "v0.4"
},
"TurnTakingMetric": {
"name": "turn_taking",
Expand Down
Loading
Loading