feat(stt): add faster-whisper provider - #2192
Conversation
GPT 5.6 Review (fork) — 🔴 changes requested (blocking)Reviewed BLOCKING -- src/kiro_crew/transcribe.py:219 -- Legitimate dictated credits are deleted
BLOCKING -- src/kiro_crew/dashboard/handlers/core.py:1285 -- Faster installation fails on native Windows x64
|
Design Review (Fable 5, fork) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS Sound provider design with real trade-off reasoning, but it puts uncancellable native inference inside the gateway process and ships PR-evidence artifacts into the tree. Watch
Suggestions
[DESIGN-REVIEWED] 70909c3 |
UX Review (Fable 5, fork) — 🟡 CONCERNSUX-level review of UX-Verdict: CONCERNS The PR removes the press-and-400 dead end for Windows-on-ARM but ships the identical dead end for pip-less/desktop-app gateways choosing Watch
Suggestions
[UX-REVIEWED] 70909c3 |
Opus 4.8 Review (fork) — ✅ no blocking findingsReviewed Review detailsI've now reviewed the full diff, including the frontend and the backend transcribe/executor/handler changes. Falsifying Candidate 1 ( The chain is real in shape —
The candidate's own note concedes it "could not confirm" a non- No other grounded defect: the No findings. [OPUS-REVIEWED] 70909c3 |
|
🤖 Kiro Crew [operator: bolichen97]: This PR has been inactive for 7+ days with failing CI. I've assessed the blockers and they appear resolvable — I'll push fixes directly to this branch as a co-author. Assessment: Merge conflict (base drift) + 2 blocking AI-review findings (hallucination filter substring-matching deletes legitimate speech; faster-whisper installs to user-site instead of venv) + 1 advisory (local imports should be top-level). All are concrete code defects with clear fixes. Build/test/lint CI is fully green on the current head. |
b772099 to
c001d68
Compare
|
🤖 Kiro Crew [operator: bolichen97] — pushed Rebase (conflict resolution)
Blocking finding 1 — hallucination filter deletes real speech (GPT + Opus + UX) Blocking finding 2 — install targets an interpreter the gateway cannot import from (GPT + Design Review)
Advisory — function-local imports (GPT) Verification: full backend pytest (50k tests; remaining failures reproduced identically on pristine main in this sandbox — environmental), isort/flake8/mypy clean, Author identity preserved (single commit, |
|
Dispositions for the GPT review of
|
|
Answering the advisory CONCERNS verdicts on Design Review — "Install ≠ usable" (interpreter mismatch, cached Design Review — executor fit + no timeout, model constructed per call — accepted-and-deferred → #3780. Memoizing the model keyed by UX Review — filter silently deletes real speech — fixed in UX Review — Windows-on-ARM install dead end — accepted-and-deferred → #3782. Surfacing the alternatives line in UX Review (suggestion) — provider description string omits the new provider — accepted-and-deferred, folded into #3782's scope note. The string is a 14-catalog change ( Local Opus advisory — mid-transcript boilerplate excision can join unrelated sentences — accepted-and-deferred → #3781 (restrict stripping to transcript edges, where Whisper actually hallucinates). |
c001d68 to
33a4ab8
Compare
|
🤖 Kiro Crew [operator: bolichen97] — pushed
No production code changed in this push (harness + evidence only, same single commit, author preserved). |
33a4ab8 to
fdca448
Compare
|
🤖 Kiro Crew [operator: bolichen97] — pushed |
First Principles Review (Fable 5, fork) — ✅ PASSPremise-level review of All base facts check out: First-Principles-Verdict: PASS Every item traces to a nameable constraint — a missing wheel, an uncancellable future, a documented Whisper artefact — and registration lands as data, not new conditionals. What this change shipsIntent: make speech-to-text work on machines where installing the whisper CLI + ffmpeg toolchain is the blocker — an ADDITION, with two declared adjacent fixes riding along.
Watch
[FIRST-PRINCIPLES-REVIEWED] 70909c3 |
1263712 to
c9e19a8
Compare
|
Answering the CONCERNS verdict on
|
c9e19a8 to
52eee22
Compare
|
Round 4 on
|
|
Round 4 on
|
|
Round 4 on
|
|
Round 4 on
|
52eee22 to
b120248
Compare
|
Drive-to-green update (automated pipeline, adopted without a triage spec — work derived from live CI state) Head is now
No code or design changes beyond conflict resolution — the diff is the author's own (+2752/−45 across 32 files, matching the pre-rebase scope). Diagnosis of the previous red run (all 6 failures): the lockdown-before-publish failure ( Local verification before push: touched backend tests (790 passed incl. the 36 new Known main-side breakage this PR will inherit on the merge ref (not caused here):
|
b120248 to
f172036
Compare
|
Drive-to-green round 2: head is now |
|
Drive-to-green status: PR-side work COMPLETE — remaining reds are all main-side breakage. Escalating with Head
What remains red on run 32855305723, each verified to reproduce on main's own push run 32845454629 at
The AI review bots (GPT/Opus/Design/UX/First-Principles) are skipped because fork review workflows gate on CI success — they will fire automatically once main is repaired and this PR is re-run. Ask for a maintainer: land the fixes for #5843 (or merge #5850), #5846, and #5888, then re-run this PR's CI (or push a trivial rebase). No PR-side action is expected to be needed beyond that. |
|
Drive-to-green round 3 — re-driving now that main is repaired. Head is All three main-side blockers named in my escalation above are now closed, so the reds that stalled this PR are no longer real:
Change in this round: the rebase only. Rebased onto current Local verification on the rebased tree: 876 backend tests pass ( The five AI review lanes (GPT/Opus/Design/UX/First-Principles) have never actually run on this PR — they gate on |
|
GPT 5.6 Review dispositions for Note first: the lane concluded 1. BLOCKING —
The mechanism is real and I am not disputing it. I am not applying the prescribed fix, because "remove text-only deletion from the transcription path" deletes one of the three features this PR exists to add, and this is a drive-to-green pass with an explicit constraint not to change the author's design or scope. It also reverses a decision the author took deliberately and documented in the diff:
So the author already ran this exact trade-off and drew the line one notch away from where the reviewer would draw it. The disagreement is not about a defect in the code, it is about whether the retained "watching"/"subscribe" entries are plausible dictated speech — and answering it decides whether the filter ships at all, ships with a shorter list, or ships only as the language-independent repetition collapse. That is a product call on someone else's feature, so I am putting it to the maintainer rather than guessing. Three coherent options, for whoever picks this up:
I have deliberately not picked one. Note also that a maintainer override is available if the judgement is that the author's line is the right one. 2. FINDING — The Switching the faster install to a direct 3. FINDING — The docstring states the opposite of what the finding assumes — the fallback is scoped to non-strings, on purpose:
Coercing unknown strings to No code changes in this round. CI is fully green on this head (all backend/frontend shards, lint, build, E2E); Opus reports no blocking findings. |
|
Status: 60/60 CI checks green, mergeable, one blocking review verdict left — escalating with Round 3 result at The single blocker is GPT's Ask for a maintainer: pick one of those three, or override the GPT lane if the author's line is the right one. Any of the three is a small change once the call is made — the shortened-list option is a few lines in |
Three related changes, ported from the abandoned feat/meetnote branch and
finished where that port stopped short.
A `faster` provider. Unlike `whisper` and `mlx` it runs in-process rather
than shelling out, and it decodes audio through PyAV's bundled FFmpeg
instead of the system binary -- so it needs neither the binary-discovery
path nor a separately installed ffmpeg. FFmpeg is not absent: it arrives
inside PyAV's wheel. What goes away is the install step and the search
for a binary, and that is the reason to have it: on a machine where
installing the CLI toolchain is the hard part, this is a pip install of
prebuilt wheels. Inference is quantised to int8 and offloaded to the
subprocess executor, since it is CPU-bound and the segment iterator does
the work as it is consumed.
Scope, since upstream has moved: `apple` now covers macOS 26+ at roughly
fifty times whisper's speed with nothing to install, so this provider
earns its place on Linux, Windows, and older macOS -- the platforms with
no `apple` path, which are also where getting ffmpeg onto the machine is
most often the obstacle.
A hallucination filter for the Whisper family. Whisper models fed silence
emit two recognisable artefacts: one phrase repeated many times, and
caption boilerplate memorised from training-set video subtitles. Both
matter more here than in a general transcriber because this text reaches
agents -- a phrase repeated forty times becomes forty note lines, and a
memorised caption credit becomes a meeting note. A transcript that filters
down to nothing returns None rather than boilerplate.
The phrase half only removes caption SELF-ATTRIBUTION ("Subtitles by
Amara.org", "Transcribed by"), never a spoken sign-off. Sign-offs and
subscribe CTAs ("Thank you for watching", "Hit the bell") were dropped
from the list during review: each is a sentence someone recording a demo
or dictating a video script genuinely says, and because the match is
whole-sentence and an emptied transcript becomes None, filtering one could
delete the only words a recording held. An un-repeated hallucinated
sign-off therefore survives into the transcript -- the safe direction of
that trade, and the repetition collapse still catches the far more common
form, where the model emits the sign-off for the rest of the window.
AWS Transcribe is excluded from both halves: it uses a different decoder
and does not produce these, so filtering it could only ever delete real
speech.
The full model enum. `stt.model` accepted exactly one value, `turbo`,
which is a strange thing for a field named "model size". It now takes
tiny/base/small/medium/large-v3/turbo -- the small end is the difference
between usable and unusable on a machine short of RAM, and large-v3 is
the accuracy ceiling. `turbo` stays the default. An unrecognised value
warns and falls back rather than raising, matching the provider
validator: a typo in one config field must not stop the Gateway booting.
Three things the port source left undone, fixed here:
- `_STT_MODEL_SIZES` is the dashboard's PUT allowlist, so expanding the
loader's enum without expanding it would have had the API silently
reject every new model. A test now pins the two sets equal.
- The settings UI had no `faster` entry at all: the dropdown would have
read a bare "faster", and the model picker was gated on `whisper`
alone, so choosing `faster` showed no model control while the backend
went on reading `stt.model`. Both fixed, with the install button, its
blurb and the progress-step label.
- `cli_doctor` reported a missing ffmpeg as an ISSUE for a provider that
never looks for one, and `_stt_prereq_commands` offered the whole
brew/Xcode/python toolchain as prerequisites for a plain pip install.
Neither applies to `faster`.
The user-facing copy says "no separate ffmpeg install", not "no ffmpeg
needed", in all eleven catalogues. The shorter claim was wrong in a way
that matters: it tells the user nothing of FFmpeg lands on their machine,
when in fact PyAV's wheel carries it. The test that pins this string now
asserts the accurate claim rather than merely that it mentions ffmpeg, so
the imprecise wording cannot come back.
faster-whisper is deliberately NOT added to setup.cfg extras. It is
installed on demand from Settings, and cli_doctor names the Windows-ARM
case where CTranslate2 publishes no wheel and no install can succeed.
Two existing tests pinned the exact provider list while testing mlx
gating; their expectations are updated, and a note says they track
_VALID_STT_PROVIDERS on purpose. The new UI test mirrors that list too,
and had gone stale against upstream's `apple` -- the omission the mirror
exists to catch, so it is fixed and commented.
Verified: 1,301 pytest passed across the stt/transcribe selection; the
faster-whisper UI and i18n parity suites pass (79 tests); isort, flake8
and mypy clean. Transcription accuracy itself is unverified --
faster-whisper is not installed here, so the library is patched in every
test.
Co-authored-by: Bolin Chen <bolichen@amazon.com>
Rebased onto current main and folded in the two review follow-ups filed
against this PR, so they land with the provider rather than after it.
Six files conflicted against a base that had moved 766 commits, and two
of those conflicts were semantic rather than textual: main added the
parakeet provider, so _VALID_STT_PROVIDERS, the config baseline and
two provider-list mirrors in the tests all needed both additions, not
either one. The i18n catalogs and the settings page merged cleanly.
Issue kirodotdev#3782 -- surface the Windows-on-ARM wheel gap. CTranslate2
publishes no win-arm64 wheel and no sdist, so pip fails while RESOLVING
and names ctranslate2, a package the user never asked for. That reads
like a transient registry problem and invites retrying something that
can never work, so �pi_stt_install now refuses up front and names the
two providers that do work here.
The check is server-side rather than inside the generated script, and
that is a correctness requirement, not a preference: the script is
launched through �ash -c, which does not exist on a stock
native-Windows gateway, so the run dies with FileNotFoundError before
executing a line. A guard inside the script would be unreachable on
exactly the platform it is for. The same verdict is served from the
config GET as aster_unsupported, mirroring ranscribe_unsupported,
so the Settings card shows the notice and the alternatives BEFORE the
press instead of turning every press into an identical 400.
platform_compat.is_windows_on_arm() deliberately reports the PROCESS
architecture. Windows on ARM runs x86-64 processes under emulation, and
such an interpreter installs the win_amd64 wheel and works -- a
host-architecture probe would refuse a package that succeeds.
Issue kirodotdev#3780 -- honor stt.timeout_secs. Its first half (memoizing the
constructed WhisperModel per model+device) was already implemented here,
so only the unbounded future remained. The inference future is now
bounded like every other provider's work, and it moved off
subprocess_executor onto a dedicated stt_executor. That pool split
is the substance: a started run_in_executor future cannot be cancelled,
so a wedged model load -- or a first-run multi-GB weight download inside
the library constructor -- holds its worker until the process exits. On
the PTY-teardown pool it would consume one of the eight workers whose
whole purpose is absorbing a teardown storm, making the recovery path
starvable by the thing it recovers from. Two workers, because each
in-flight call keeps a fully quantised model resident, so the count is a
memory ceiling and not only a CPU one. The timeout releases the CALLER,
never the thread, and both the log line and the docstring say so rather
than implying a cancel.
One judgement call worth naming: cli_doctor reports the win-arm case as
a hard issue even on Windows, departing from the stt_fatal downgrade
its neighbours use. That downgrade exists because whisper and ffmpeg are
absent from a stock Windows box yet trivially installable, so failing a
first-run doctor over them is noise. This is the opposite case --
aster is never the default, so reaching it means the user explicitly
chose a provider that cannot be made to work -- and a run that exits 0
would contradict the alternatives it just printed. The ordinary
not-installed arm still follows the convention, because the install
button does fix that one.
Verified on Windows: 828 passed / 52 skipped across the stt, transcribe,
dashboard-core, cli, platform-compat, executors and config-baseline
suites, plus 13 frontend tests for the notice and its catalogs. flake8,
isort, black (diff-scoped gate) and mypy --platform linux all clean --
mypy went from 7 errors to 6, the fixed one being a dangling reference
this rebase would otherwise have shipped. One inherited failure,
test_executors.py::test_cancelling_the_caller_does_not_kill_a_job_already_running,
reproduces identically on unmodified main. Transcription accuracy stays
unverified: faster-whisper is not installed here, so the library is
patched in every test.
Co-authored-by: chenmingwei23 <chenmingwei98@gmail.com>
Co-authored-by: Kiro Crew <noreply@kirodotdev.github.io>
|
Round 4 — the GPT blocking finding is now FIXED, not deferred. Head The maintainer call came back as "shorten the list", so the What changed (2 files, no behaviour outside the filter):
The list rule itself is what actually moved. It was "implausible as a complete DICTATED utterance", and that bar is what admitted the deleted entries — it is a judgement call, and it went the wrong way on exactly the sentence the finding names. The new bar is mechanical: an entry must be text that exists because a transcript was produced, not because anyone spoke. Nobody utters "Subtitles by the Amara.org community" into a voice memo, so no reading of the surviving list deletes speech. Residual, stated rather than hidden: a single un-repeated hallucinated sign-off now survives into the transcript. That is the safe direction of the trade — one stray line a reader can see and ignore, versus silently destroying the only words a recording held — and the repetition collapse still removes the far more common form of the artefact, where the model emits the sign-off for the remainder of the decode window. The repetition half is untouched, and it is the language-independent half. Tests — the list discipline is now pinned in BOTH directions, because the required-marker test alone was never the guard: it passes for a phrase like "subscribe to my subtitles too". So the sign-off vocabulary is banned outright as a second invariant.
Mutation-verified: re-adding Also in this push: rebased onto current Verification: 982 backend tests pass ( The two other GPT findings from this lane keep their earlier dispositions: the |
|
Round 4 verdict read on 1. BLOCKING —
This is the third consecutive round on this one span, and the third time the prescribed fix is "delete the feature." The bar moved each round rather than the code being wrong:
That second premise is where I stop agreeing. The match is whole-sentence after punctuation stripping, so every ordinary use of those words is already safe and is pinned by tests: "The transcript is available in the shared drive", "We kept transcribed by in the caption doc", "Thanks for joining, everyone." all survive. For the finding to bite, a user must dictate a bare caption credit as their entire utterance — and the ten survivors are all grammatically incomplete fragments ( Against that residual sits the reason the filter exists: this text goes to agents, and a memorised caption credit becomes a meeting note. Deleting the phrase half would also make the filter English-repetition-only, which is a product regression, not a hardening. Per the review loop's own stop rule — same 2. BLOCKING — The mechanism is pre-existing and shared: What is fair, and why I am not simply re-rebutting: this PR's body claims Windows as a target platform for the provider ("earns its place on Linux, Windows, and older macOS"). A promise the install path cannot keep is the PR's own defect even when the broken path is inherited. So it needs a call, and the options differ in scope enough that I should not pick for you:
3. Advisory lanes. First Principles: CI itself is green across every shard, lint gate, build, packaging and E2E on this head, and the PR is |
Problem / Motivation
The Whisper-family providers need two things from the machine before they work: the
whisper(ormlx_whisper) CLI on disk somewhere discoverable, and a systemffmpegbinary. On a machine where getting that toolchain installed is the hardpart — no Homebrew, no Xcode CLT, a locked-down Linux box, Windows without a package
manager — speech-to-text is simply unavailable, and the Settings panel can only
offer to install a toolchain the user cannot install.
Two smaller gaps in the same area:
stt.modelaccepted exactly one value,turbo. For a field named "modelsize" that is a strange constraint, and it is the wrong one on a machine short of
RAM, where the small end is the difference between usable and unusable.
A phrase repeated forty times becomes forty note lines; a memorised caption
credit ("Subtitles by Amara.org") becomes a meeting note.
Why it matters
STT is the input to everything the meetings app does. When it is unavailable there
is no transcript, so there are no minutes, no summary and no action items — and the
reason is an install step, not a missing feature.
The hallucination case is worse than noise: it is plausible text. A user reading
minutes cannot tell a memorised caption from something someone said.
What changed (motivation → approach → change)
Approach. Add a provider that removes the two prerequisites instead of trying to
install them, and fix the two adjacent gaps while the provider list is being touched
anyway.
The
fasterproviderfaster-whisperruns the same Whisper weights on a different inference engine(CTranslate2). It is not a different or better model — what changes is how it runs:
So it needs neither the binary-discovery path nor a separately installed ffmpeg.
To be precise about what lands on the machine: FFmpeg is not absent — it arrives
inside PyAV's wheel, and faster-whisper decodes through it in-process. What goes
away is the install step and the search for a binary. On a machine where the CLI
toolchain is the obstacle, this is a pip install of prebuilt wheels.
Inference is quantised to int8 — that is what makes CPU inference practical — and
offloaded to the subprocess executor, since it is CPU-bound and the segment iterator
does the work as it is consumed.
Scope, since upstream has moved.
applenow covers macOS 26+ at roughly fiftytimes whisper's speed with nothing to install (measured on this machine: 86.6s of
audio in 0.61s). So this provider earns its place on Linux, Windows, and older
macOS — the platforms with no
applepath, which are also where getting ffmpegonto the machine is most often the problem. It is not proposed as the macOS default.
The hallucination filter
Applied to the Whisper family only. Two artefacts are removed: a phrase repeated in
a consecutive run, and caption boilerplate memorised from training-set video
subtitles. A transcript that filters down to nothing returns
Nonerather thanboilerplate.
The phrase half removes caption SELF-ATTRIBUTION only — "Subtitles by
Amara.org", "Transcribed by", and the like — never a spoken sign-off. Sign-offs and
subscribe CTAs ("Thank you for watching", "Hit the bell") were on the list and have
been removed after review, because each is a sentence someone recording a demo or
dictating a video script genuinely says. The match is whole-sentence and an emptied
transcript becomes
None, so filtering one of those could delete the only words arecording held. The consequence is deliberate and stated: a single un-repeated
hallucinated sign-off now survives into the transcript. That is the safe direction
of the trade — a stray line a reader can see and ignore, rather than silently
destroying speech — and the repetition collapse still catches the far more common
form of the artefact, where the model emits the sign-off for the remainder of the
decode window.
AWS Transcribe is excluded from both halves deliberately: it uses a different
decoder and does not produce these, so filtering it could only ever delete real
speech.
The full model enum
tiny/base/small/medium/large-v3/turbo, withturbostill the default.An unrecognised value warns and falls back rather than raising, matching the provider
validator — a typo in one config field must not stop the Gateway booting.
Three things the port source left undone
This started as a port from an abandoned branch; these are the parts that port
stopped short of, and each is a silent failure rather than an error:
_STT_MODEL_SIZESis the dashboard's PUT allowlist. Expanding the loader's enumwithout expanding it would have had the API silently reject every new model. A
test now pins the two sets equal.
fasterentry at all: the dropdown would have rendered abare
"faster", and the model picker was gated onwhisperalone — so choosingfastershowed no model control while the backend went on readingstt.model.cli_doctorreported a missing ffmpeg as an ISSUE for a provider that never looksfor one, and
_stt_prereq_commandsoffered the whole brew/Xcode/python toolchainas prerequisites for a plain pip install. Neither applies to
faster.Not added to
setup.cfgextras, deliberately. It is installed on demand fromSettings.
cli_doctornames the Windows-on-ARM case, where CTranslate2 publishes nowheel and no install can succeed, rather than letting it fail as a generic pip error.
Tests
test/test_transcribe_faster.py— 36 tests, no gateway or model needed; thelibrary is patched throughout.
fasteris in the Whisper family whiletranscribeis notturbostill default, an unknown value fallsback instead of raising, and the dashboard allowlist covers every valid model
the property the whole provider rests on
becomes
Nonespeech survives intact
and no entry carries the vocabulary of a spoken sign-off — plus a regression test
that each removed sign-off now survives as a whole transcript (mutation-verified:
re-adding one phrase fails four tests)
Windows-ARM gap, and emits the progress line the status parser matches
website/src/test/SttSettingsFaster.test.ts— 12 tests pinning the three mapsthat are the UI's contract with the backend's vocabularies. All three are silent
failures if wrong, which is why they are pinned rather than left to a screenshot:
a provider missing from
PROVIDER_LABEL_KEYrenders as a bare id, a step missingfrom
STEP_LABEL_KEYrenders as an empty progress label, and a provider missingfrom
WHISPER_MODEL_PROVIDERSgets no model picker.One of those is a mirror of the backend's
_VALID_STT_PROVIDERS, maintained by handbecause that list is Python. It had gone stale against
apple— the exact omissionthe mirror exists to catch — so it is fixed and commented.
The copy test asserts the accurate claim ("no separate ffmpeg install", and that
PyAV is named) rather than merely that the string mentions ffmpeg, so the imprecise
wording cannot come back.
Manual verification
Transcription accuracy is not verified. faster-whisper is not installed in this
environment, so the library is patched in every test. What is verified is everything
around it: that the provider is registered and dispatched, that it never reaches the
ffmpeg probe or a subprocess, that the install script does what its progress parser
expects, and that the UI offers the provider with a model picker and an install
button.
Checked on this machine: 34,809 pytest passed with no STT-family failure; the whole
website suite green (825 files, 10,963 tests); isort, flake8, mypy and
tsc -bclean; eslint 0 errors; the pseudolocale gate regenerated and matching.
Screenshots / video
The dropdown, the model picker and the install progress are being captured and will
be added here shortly — three stills:
faster-whisperas an option
faster-whisperselected, showing its blurb and the model picker that onlyappears for Whisper-size providers
Installing faster-whisper…progress stepHappy for code review to start in the meantime — the UI change is confined to
SttSettings.tsxand is pinned by the 12 tests above.Related Issues
N/A — found while getting STT working on a machine without the CLI toolchain, not
from a filed issue.
Checklist
Settings and in
cli_doctor, which is where a user meets itScreenshots
Captured with the committed harness
website/scripts/capture-stt-faster.mjs(real built SPA against fixture APIs; each frame's strings are asserted programmatically before the shot).Provider selected, library not yet installed — localized dropdown label, the model picker now gated on
WHISPER_MODEL_PROVIDERS(previouslywhisperalone), and the Install button with its "no separate ffmpeg install" blurb:Install in flight + ready state