diff --git a/configs/prompts/judge.yaml b/configs/prompts/judge.yaml index e4b4ba14..b29949dc 100644 --- a/configs/prompts/judge.yaml +++ b/configs/prompts/judge.yaml @@ -1071,3 +1071,169 @@ judge: "summary": "<1-2 sentence summary for this turn>" }} ] + + audio_native_prompt: | + You are an expert evaluator analyzing whether an AI voice agent correctly **understood** the key entities a user provided during a spoken conversation. + + This is a speech-to-speech / audio-native agent: there is no reliable speech-to-text transcript of the user to inspect. Instead, the evidence of what the agent actually understood is what it did with the information — primarily the **arguments it passed to its tool calls**, and secondarily any values it read back to the user. When a user says "my id is EM123" and the agent then calls a tool with "EMP124", the agent misheard the entity. + + Your task: + 1. For EACH user turn, identify all key entities in what the USER SAID (the "User said" text). + 2. For each entity, find where the agent used it — a tool call argument (see the Conversation below) or a value the agent read back — and check whether the agent captured it CORRECTLY. + 3. Mark each entity as correct or incorrect. + 4. If the agent never used a given entity anywhere (no tool call argument and no read-back reference to it), there is no evidence to judge understanding — include the entity but mark it as skipped. + 5. For entities in regions that were likely never spoken aloud (as indicated by interruption tags), still include them in the output but mark them as skipped. + + ## What Counts as an Entity + An entity must have a **specific, concrete value** — something that could be passed as an input to a program or tool (not an AI, but a script or database lookup). Ask yourself: could this value be stored in a variable and used programmatically? + + - Names (people, places, organizations): e.g. "John Smith", "Austin", "Delta Airlines" + - Specific dates and times: e.g. "December 15th", "3:45 PM" — NOT vague references like "tomorrow morning" or "later today" + - Confirmation codes / reference numbers: e.g. "ABC123", "ZK3FFW" + - Flight numbers: e.g. "UA 204" + - Amounts and prices: use the specific value only, e.g. "$120" — for qualifier phrases like "under $120", only use the specific value + - Addresses: e.g. "123 Main Street" + - Phone numbers: e.g. "555-867-5309" + - Email addresses: e.g. "john@example.com" + - Other specific identifiers: seat numbers, loyalty numbers, booking IDs, employee IDs, etc. + + **Not an entity:** vague temporal words ("tomorrow", "next week", "morning"), general descriptors ("the cheap flight", "a long trip"), or open-ended qualifiers ("less than an hour", "around noon"). + + ## IMPORTANT: What This Metric Measures + This measures whether the agent **understood** the entity the user provided, as evidenced by how it used it. It is NOT a faithfulness or tool-correctness metric. Do NOT penalize: + - The agent choosing the wrong tool, or omitting a tool call it should have made + - The agent using an entity that came from a tool RESPONSE rather than from the user (only user-provided entities are evaluated here) + - Correct values that simply never got reused (mark those skipped, not incorrect) + + Only mark an entity incorrect when the agent demonstrably used a **different value** than the user provided (e.g. user "EM123" → tool call "EMP124", user "March 25th" → tool call "2026-03-26"). + + ## Understanding Tags in the User Text + The user text may contain non-spoken tags and markers. These are metadata — they were never said aloud and must not be treated as entities or evaluated. + + ### Audio-Direction Tags + Tags like [slow], [firm], [annoyed] describe how the words were meant to be spoken. Ignore them entirely. + + ### Interruption Tags + {interruption_tags_reference} + + These markers indicate that parts of the user text may never have been spoken aloud, because the user was interrupted or talked over. An entity that was never spoken cannot have been understood, so you must NOT penalize for entities in regions that were likely not said — mark them as skipped. + + **Key principle:** Only evaluate entities that were reasonably expected to have been spoken aloud AND that the agent had an occasion to use. If a tag indicates the user was interrupted before or during an entity, still include the entity but set `skipped: true` and explain why. + + ## User Turns to Evaluate + {user_turns} + + ## Conversation (with agent turns and tool calls/responses) + Use this to find where each user-provided entity was used by the agent. Tool call arguments are the primary evidence of what the agent understood. + {conversation} + + ## Correctness Criteria + - Entity must have been used correctly by the agent — the value passed to the tool call (or read back) must match what the user said. + - Numbers: "150" and "one hundred fifty" are equivalent + - Dates: "December 15th" and "Dec 15" are equivalent + - Names: Case-insensitive match. Phonetically equivalent spelling variants are acceptable on the **first mention** of a name (e.g., German Meyer/Meier/Maier/Mayer — all sound the same when spoken). Once the user has explicitly disambiguated the spelling (e.g., spelled it out letter by letter, corrected a prior spelling, or confirmed a specific spelling after the assistant read it back), subsequent usage must match that clarified spelling exactly. Flag if: + - The value the agent used clearly differs (different sound, not just different spelling) + - The agent used a different spelling variant **after** the user has explicitly clarified the correct spelling + - Non-Latin scripts: A name or identifier rendered in its native writing system (e.g. Kanji, Hangul, Arabic) is equivalent to its romanized/phonetic form if they represent the same spoken entity. For example, "山田" and "Yamada" should be treated as matching. + - Phonetic letter spellings: In languages that do not use the Latin alphabet, Roman/Latin letters are spoken using language-specific phonetic equivalents. These are correct, not errors (e.g., in Korean "에이" for "A" or "비" for "B"). + - Number surface variants: Many languages have systematic spoken-form differences that do not change the underlying value and must not be penalized: + - Reversed digit order in spoken form (e.g., German "vierundzwanzig" = four-and-twenty = 24) + - Grammatical gender/agreement variants (e.g., Spanish "veintiún" vs "veintiuna" for 21) + - Large-unit grouping differences (e.g., Japanese 万-system: "ichi-man go-sen" / 一万五千 = 15,000) + - Honorific/title variants: "Ms." vs "Miss", "Mr." vs "Mister", and language equivalents (e.g., French "Madame" vs "Mme", Japanese "様" vs "-san") are acceptable variants of the same title. + + Important note: The user text will often feature things formatted like "one two three" instead of "123". Your goal is to evaluate semantic equivalence — these are considered equivalent if the value matches what the agent used. + + ## Examples + + **Example Input:** + User Turns to Evaluate: + Turn 1: + User said: `My employee id is E M one two three and my name is John Smith.` + + Turn 2: + User said: `Book it for December 15th.` + + Turn 3: + User said: `[slow] The code is X X F six O H, with the letter O, [assistant interrupts] not zero.` + + Conversation: + Turn 1: + user: My employee id is E M one two three and my name is John Smith. + tool_call: lookup_employee({{'employee_id': 'EMP124'}}) + Turn 2: + user: Book it for December 15th. + tool_call: create_booking({{'date': '2026-12-15'}}) + + **Example Response:** + [ + {{ + "turn_id": 1, + "entities": [ + {{ + "type": "employee_id", + "value": "E M one two three", + "captured_value": "EMP124", + "analysis": "User said EM123 but the agent called lookup_employee with EMP124 — misheard.", + "correct": false, + "skipped": false + }}, + {{ + "type": "name", + "value": "John Smith", + "captured_value": "not used", + "analysis": "The name was never passed to a tool call or read back, so there is no evidence of how the agent understood it.", + "correct": false, + "skipped": true + }} + ], + "summary": "1 entity used (employee_id, incorrect). Name never used — skipped." + }}, + {{ + "turn_id": 2, + "entities": [ + {{ + "type": "date", + "value": "December 15th", + "captured_value": "2026-12-15", + "analysis": "Agent passed 2026-12-15 to create_booking — matches December 15th.", + "correct": true, + "skipped": false + }} + ], + "summary": "1 entity correct." + }}, + {{ + "turn_id": 3, + "entities": [ + {{ + "type": "confirmation_code", + "value": "X X F six O H", + "captured_value": "not used", + "analysis": "Code appears before the [assistant interrupts] tag, but the agent never used it in a tool call — no evidence to judge, skipped.", + "correct": false, + "skipped": true + }} + ], + "summary": "1 entity found; never used by the agent — skipped." + }} + ] + + ## Response Format + Respond with a JSON object. Each turn entry must include the turn_id matching the turn number shown in the User Turns to Evaluate section above: + [ + {{ + "turn_id": , + "entities": [ + {{ + "type": "", + "value": "", + "captured_value": "", + "analysis": "", + "correct": , + "skipped": + }} + ], + "summary": "<1-2 sentence summary for this turn>" + }} + ] diff --git a/src/eva/__init__.py b/src/eva/__init__.py index eb2eaa6a..5a7690b0 100644 --- a/src/eva/__init__.py +++ b/src/eva/__init__.py @@ -11,4 +11,4 @@ # Bump metrics_version when changes affect metric computation (metrics code, # judge prompts, pricing tables, postprocessor). -metrics_version = "2.2.0" +metrics_version = "2.3.0" diff --git a/src/eva/metrics/diagnostic/transcription_accuracy_key_entities.py b/src/eva/metrics/diagnostic/transcription_accuracy_key_entities.py index 8647d7f4..806974ea 100644 --- a/src/eva/metrics/diagnostic/transcription_accuracy_key_entities.py +++ b/src/eva/metrics/diagnostic/transcription_accuracy_key_entities.py @@ -1,4 +1,16 @@ -"""Transcription accuracy key entities metric using LLM-as-judge (entire conversation). +"""Key entity accuracy metric using LLM-as-judge (entire conversation). + +Measures whether the user's key entities (names, dates, confirmation codes, +amounts, etc.) survived intact through the agent's pipeline. The comparison +target depends on the pipeline type: + +- **CASCADE**: compares what the user was instructed to say (``intended_user_turns``) + against what the agent's STT transcribed (``transcribed_user_turns``) — i.e. + transcription accuracy. +- **Audio-native (S2S / AUDIO_LLM)**: there is no reliable STT output to inspect, + so the agent's **tool-call arguments** (from ``conversation_trace``) become the + evidence of what it understood. If the user says "my id is EM123" and the agent + then calls a tool with "EMP124", the agent misheard the entity. Debug metric for diagnosing model performance issues, not directly used in final evaluation scores. @@ -10,23 +22,29 @@ from eva.metrics.registry import register_metric from eva.metrics.utils import ( aggregate_per_turn_scores, + format_transcript_with_tools, make_rate_sub_metric, parse_judge_response_list, resolve_turn_id, ) -from eva.models.config import PipelineType from eva.models.results import MetricScore @register_metric class TranscriptionAccuracyKeyEntitiesMetric(TextJudgeMetric): - """LLM-based transcription accuracy metric for key entities only (entire conversation). + """LLM-based key entity accuracy metric (entire conversation). - Evaluates STT transcription accuracy by comparing key entities (names, dates, - confirmation codes, amounts, etc.) between what the user was supposed to say - (intended_user_turns) and what STT transcribed (transcribed_user_turns). + Evaluates whether the user's key entities (names, dates, confirmation codes, + amounts, etc.) survived intact through the agent's pipeline. The comparison + target depends on the pipeline type: - Computes the ratio of correctly transcribed entities per turn: + - **CASCADE**: compares ``intended_user_turns`` against ``transcribed_user_turns`` + (STT output) — transcription accuracy. + - **Audio-native (S2S / AUDIO_LLM)**: no reliable STT output exists, so the agent's + tool-call arguments (from ``conversation_trace``) are used as the evidence of + what it understood. + + Computes the ratio of correctly captured entities per turn: score = correct_entities / total_entities Entity types evaluated: @@ -48,11 +66,12 @@ class TranscriptionAccuracyKeyEntitiesMetric(TextJudgeMetric): """ name = "transcription_accuracy_key_entities" - version = "v0.3" - description = "Debug metric: LLM judge evaluation of STT key entity transcription accuracy for entire conversation" + version = "v0.4" + description = ( + "Debug metric: LLM judge evaluation of user key entity accuracy (STT in cascade, tool calls in audio-native)" + ) category = "diagnostic" exclude_from_pass_at_k = True - supported_pipeline_types = frozenset({PipelineType.CASCADE}) rating_scale = None # Custom scoring (not 1-3 scale) default_aggregation = "mean" @@ -70,9 +89,7 @@ async def compute(self, context: MetricContext) -> MetricScore: MetricScore with aggregated score and per-turn details """ try: - turns_to_evaluate = self._get_turns_to_evaluate(context) - user_turns_text = self._format_user_turns(turns_to_evaluate, context) - prompt = self.get_judge_prompt(user_turns=user_turns_text) + prompt, turns_to_evaluate = self._build_prompt_and_turns(context) response_text = await self._call_judge_raw(prompt, context) turn_evaluations = parse_judge_response_list(response_text) @@ -206,14 +223,43 @@ def _build_per_entity_type_sub_metrics( ) return sub_metrics + def _build_prompt_and_turns(self, context: MetricContext) -> tuple[str, list[int]]: + """Select the comparison source and judge prompt based on pipeline type. + + CASCADE compares intended vs STT-transcribed user turns. Audio-native + pipelines (S2S / AUDIO_LLM) have no reliable STT, so the user's intended + turns are compared against the agent's tool-call arguments (from + ``conversation_trace``), which are the evidence of what it understood. + + Returns: + Tuple of (rendered judge prompt, sorted turn IDs to evaluate). + """ + if context.is_audio_native: + turn_ids = self._get_audio_native_turns(context) + prompt = self.get_judge_prompt( + prompt_key="audio_native_prompt", + user_turns=self._format_intended_user_turns(turn_ids, context), + conversation=format_transcript_with_tools(context.conversation_trace), + ) + return prompt, turn_ids + + turn_ids = self._get_turns_to_evaluate(context) + prompt = self.get_judge_prompt(user_turns=self._format_user_turns(turn_ids, context)) + return prompt, turn_ids + @staticmethod def _get_turns_to_evaluate(context: MetricContext) -> list[int]: - """Return sorted turn IDs present in both tts_text_user and transcript_user.""" + """Return sorted turn IDs present in both intended and transcribed user turns.""" return sorted(context.intended_user_turns.keys() & context.transcribed_user_turns.keys()) + @staticmethod + def _get_audio_native_turns(context: MetricContext) -> list[int]: + """Return sorted intended user turn IDs (ground truth for audio-native pipelines).""" + return sorted(context.intended_user_turns.keys()) + @staticmethod def _format_user_turns(turn_ids: list[int], context: MetricContext) -> str: - """Format user turns for the prompt.""" + """Format expected-vs-transcribed user turn pairs for the cascade prompt.""" return "\n\n".join( f"Turn {tid}:\n" f'Expected: "{context.intended_user_turns[tid]}"\n' @@ -221,6 +267,11 @@ def _format_user_turns(turn_ids: list[int], context: MetricContext) -> str: for tid in turn_ids ) + @staticmethod + def _format_intended_user_turns(turn_ids: list[int], context: MetricContext) -> str: + """Format the user's intended utterances for the audio-native prompt.""" + return "\n\n".join(f'Turn {tid}:\nUser said: "{context.intended_user_turns[tid]}"' for tid in turn_ids) + async def _call_judge_raw(self, prompt: str, context: MetricContext) -> str | None: """Call LLM judge and return raw response text. diff --git a/tests/fixtures/metric_signatures.json b/tests/fixtures/metric_signatures.json index 3b95ed18..280ba360 100644 --- a/tests/fixtures/metric_signatures.json +++ b/tests/fixtures/metric_signatures.json @@ -86,8 +86,8 @@ "TranscriptionAccuracyKeyEntitiesMetric": { "name": "transcription_accuracy_key_entities", "prompt_hash": "877dfa61f232", - "source_hash": "70e92b2f7221", - "version": "v0.3" + "source_hash": "1be843cc964d", + "version": "v0.4" }, "TurnTakingMetric": { "name": "turn_taking", diff --git a/tests/unit/metrics/test_transcription_accuracy_key_entities.py b/tests/unit/metrics/test_transcription_accuracy_key_entities.py index 654c7ed7..d8de0216 100644 --- a/tests/unit/metrics/test_transcription_accuracy_key_entities.py +++ b/tests/unit/metrics/test_transcription_accuracy_key_entities.py @@ -9,6 +9,7 @@ TranscriptionAccuracyKeyEntitiesMetric, ) from eva.metrics.utils import aggregate_per_turn_scores +from eva.models.config import PipelineType from .conftest import make_judge_metric, make_metric_context @@ -375,3 +376,91 @@ async def test_no_response_from_judge(self, metric): assert result.error == "No response from judge" assert result.score == 0.0 + + +class TestPipelineSupport: + def test_supports_cascade_and_audio_native(self): + """Metric supports cascade and both audio-native pipeline types.""" + supported = TranscriptionAccuracyKeyEntitiesMetric.supported_pipeline_types + assert PipelineType.CASCADE in supported + assert PipelineType.S2S in supported + assert PipelineType.AUDIO_LLM in supported + + +class TestAudioNativeCompute: + """Audio-native pipelines compare intended user entities against tool-call arguments.""" + + @pytest.mark.asyncio + async def test_uses_audio_native_prompt_and_intended_turns(self, metric): + """Audio-native path builds the tool-call prompt from intended turns + conversation trace.""" + context = make_metric_context( + pipeline_type=PipelineType.S2S, + intended_user_turns={1: "My employee id is E M one two three"}, + transcribed_user_turns={}, # unreliable / absent for audio-native + conversation_trace=[ + {"role": "user", "content": "My employee id is E M one two three", "turn_id": 1}, + { + "type": "tool_call", + "tool_name": "lookup_employee", + "parameters": {"employee_id": "EMP124"}, + "turn_id": 1, + }, + ], + ) + response = _make_judge_response( + [ + { + "turn_id": 1, + "summary": "Misheard employee id", + "entities": [ + {"type": "employee_id", "value": "EM123", "correct": False, "skipped": False}, + ], + } + ] + ) + generate_text = AsyncMock(return_value=(response, None)) + metric.llm_client.generate_text = generate_text + + result = await metric.compute(context) + + assert result.error is None + assert result.score == 0.0 + assert result.normalized_score == 0.0 + # Prompt must be the audio-native variant fed with tool-call evidence, not STT. + sent_prompt = generate_text.call_args[0][0][0]["content"] + assert "tool call" in sent_prompt.lower() + assert "EMP124" in sent_prompt + assert "Transcribed:" not in sent_prompt + + @pytest.mark.asyncio + async def test_correct_entity_in_tool_call_scores_one(self, metric): + """Entity correctly captured in the tool call → score 1.0.""" + context = make_metric_context( + pipeline_type=PipelineType.AUDIO_LLM, + intended_user_turns={1: "Book it for December 15th"}, + conversation_trace=[ + {"role": "user", "content": "Book it for December 15th", "turn_id": 1}, + { + "type": "tool_call", + "tool_name": "create_booking", + "parameters": {"date": "2026-12-15"}, + "turn_id": 1, + }, + ], + ) + response = _make_judge_response( + [ + { + "turn_id": 1, + "summary": "Date captured correctly", + "entities": [{"type": "date", "value": "December 15th", "correct": True, "skipped": False}], + } + ] + ) + metric.llm_client.generate_text = AsyncMock(return_value=(response, None)) + + result = await metric.compute(context) + + assert result.error is None + assert result.score == 1.0 + assert result.normalized_score == 1.0