Skip to content

Extend key entities metric to audio-native models#180

Draft
weiz9 wants to merge 1 commit into
mainfrom
pr/wz/key-entities-audio-native
Draft

Extend key entities metric to audio-native models#180
weiz9 wants to merge 1 commit into
mainfrom
pr/wz/key-entities-audio-native

Conversation

@weiz9

@weiz9 weiz9 commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

What

Extends the existing transcription_accuracy_key_entities metric to support audio-native pipelines (S2S / AUDIO_LLM). It stays a single text-to-text judge — no new metric class.

Previously the metric was CASCADE-only: it compared what the user was instructed to say (intended_user_turns) against what the agent's STT transcribed (transcribed_user_turns). Audio-native models expose no reliable STT transcript, so there was nothing to compare against.

How

The metric now branches on pipeline type:

  • CASCADE (unchanged): compares intended_user_turns vs transcribed_user_turns (STT output) — transcription accuracy.
  • Audio-native (new): compares the user's intended entities against the agent's tool-call arguments (from conversation_trace), which are the evidence of what the model actually understood. If the user says "my id is EM123" and the agent calls a tool with "EMP124", the entity is flagged as misheard.

Both paths reuse the same per-entity ratio scoring (correct / total) and per-entity-type sub-metrics. Audio-native uses a new audio_native_prompt judge template; the cascade user_prompt is untouched.

Changes

  • transcription_accuracy_key_entities.py: removed the CASCADE-only restriction (now all pipeline types), added _build_prompt_and_turns() to select source + prompt via context.is_audio_native, plus _get_audio_native_turns() / _format_intended_user_turns(). Bumped metric version v0.3 → v0.4.
  • configs/prompts/judge.yaml: added audio_native_prompt under the same metric key.
  • Tests: added TestPipelineSupport and TestAudioNativeCompute (misheard → 0, correct → 1, verifies the audio-native prompt is used with tool-call evidence).
  • Bumped metrics_version 2.2.0 → 2.3.0 and regenerated metric_signatures.json.

Verification

  • All 423 metrics unit tests pass, including the drift/signature test.
  • ruff check + format clean.

Notes

  • Signature coverage gap: the drift signature only hashes judge.{name}.user_prompt, so future text edits to the new audio_native_prompt won't auto-trip the drift test — they'd need a manual version bump. Happy to extend signatures.py to hash all prompt keys if preferred.
  • Pre-existing mypy errors in this file (MetricScore pydantic defaults under strict) are unchanged; the new code adds none.

Extend transcription_accuracy_key_entities to support S2S / AUDIO_LLM
pipelines. Audio-native models expose no reliable STT transcript, so the
agent's tool-call arguments (from conversation_trace) become the evidence
of what it understood: if the user says "my id is EM123" and the agent
calls a tool with "EMP124", the entity is flagged as misheard.

The metric now branches on pipeline type:
- CASCADE: compares intended vs transcribed user turns (unchanged).
- Audio-native: compares intended user entities vs tool-call arguments,
  using a new audio_native_prompt judge template.

Bumps the metric to v0.4, metrics_version to 2.3.0, and regenerates the
metric signature fixture.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant