Skip to content

fix: launch the streaming speech helper with --fast (#5896) - #5907

Merged
bolichen97 merged 1 commit into
mainfrom
fix/apple-speech-streaming-fast-5896
Aug 30, 2026
Merged

fix: launch the streaming speech helper with --fast (#5896)#5907
bolichen97 merged 1 commit into
mainfrom
fix/apple-speech-streaming-fast-5896

Conversation

@CrysisDeu

Copy link
Copy Markdown
Collaborator

Summary

With Apple Speech (on-device) streaming and "Auto-submit when I finish speaking" enabled, partial transcripts render live but the utterance is never auto-submitted — recording stays active until manually stopped (#5896).

Root cause

StreamingSession.start() built the streaming helper argv without --fast. The bundled StreamTranscribe.swift helper only inserts .frequentFinalization into the DictationTranscriber reporting options when --fast is passed; without it the transcriber emits only volatile partials while the audio stream is open and never produces mid-stream final segments. The dashboard's semantic endpointer (_Endpointer in stt_stream.py) schedules its utterance-complete judgment exclusively from note_final(), so with zero mid-stream finals auto-submit can never fire.

Fix

Append --fast to the streaming helper argv (a bare switch — verified against the helper's own parser). The one-shot batch path at transcribe() deliberately keeps omitting the flag: its final arrives when the stream closes, and frequent finalization can trade accuracy for latency it has no use for.

Frequent finals are safe for the consumers: the stt_stream relay forwards each final and calls note_final() per event, and the frontend already accumulates multiple finals per session (the AWS Transcribe streaming path emits them the same way).

Verification

This is a macOS-only code path with no CI runner coverage for the live helper, so the spawned argv is the verifiable contract:

  • test_streaming_argv_includes_fast pins --fast in the streaming spawn argv (and that it did not swallow a neighbouring option's value).
  • test_one_shot_argv_excludes_fast pins its absence on the batch path.
  • test_swift_helper_still_accepts_fast pins the other side of the cross-process contract by source inspection: the Swift helper still parses case "--fast":, so a Swift-side rename cannot silently break every green Python test.
  • Mutation-verified: removing the flag makes the streaming pin fail.

Behavioral evidence is the reporter's consistent repro on main 6ed29f5cb (partials live, no finals, no auto-submit); the non-streaming one-shot path was confirmed unaffected.

Local gates: black/subprocess-encoding/isort/flake8/mypy clean; full pytest 68281 passed (17 failures reproduce identically on unmodified main or pass in isolation — host-env, none reachable from this diff). Pre-push reviews: GPT lane PASS (0 findings), Opus lane PASS (0 blocking, 2 advisories: Swift-side pin adopted; a fast= constructor escape hatch declined as speculative configurability for a single-caller path).

Closes #5896

@CrysisDeu
CrysisDeu requested a review from a team as a code owner August 25, 2026 16:19
@CrysisDeu
CrysisDeu requested a review from dwu96 August 25, 2026 16:19
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Aug 25, 2026
@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed ae9f5e94fdebcf545db27a50234fa909c343598f — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] ae9f5e9

Verdict parsed from the review's SHA-scoped output markers for commit ae9f5e94fdebcf545db27a50234fa909c343598f.

False positive or not applicable? A repository writer can comment:
/ai-review override fable ae9f5e94fdebcf545db27a50234fa909c343598f: <one-sentence reason>

@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of ae9f5e94fdebcf545db27a50234fa909c343598f — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

All claims verified: the Swift helper's --fast case inserts .frequentFinalization (StreamTranscribe.swift:231), the dashboard endpointer schedules exclusively from note_final() (stt_stream.py), and the streaming argv previously omitted the flag — so the fix targets the actual root cause, is minimal, and the batch path's exclusion is deliberate and tested. No design-level issues.

Design-Verdict: PASS

Root-cause one-flag fix at the argv seam, verified against both sides of the cross-process contract; nothing to change.

[DESIGN-REVIEWED] ae9f5e9

@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of ae9f5e94fdebcf545db27a50234fa909c343598f and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] ae9f5e9

False positive or not applicable? A repository writer can comment:
/ai-review override gpt ae9f5e94fdebcf545db27a50234fa909c343598f: <one-sentence reason>

@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of ae9f5e94fdebcf545db27a50234fa909c343598f — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

All claims verified. The change is a one-line argv fix plus three tests; the --fast flag pre-exists in StreamTranscribe.swift, the batch helper is a separate Swift source that never parses it, and the streaming helper has exactly one production spawn site. Final review:

First-Principles-Verdict: PASS

A reported defect (#5896), fixed at the mechanism — the argv that chose the wrong reporting options — with the flag's cross-process contract pinned.

What this change ships

Intent: make Apple on-device streaming dictation auto-submit when the speaker stops. FIX.

  1. Auto-submit now fires on the Apple streaming path — justified (reported defect Apple Speech streaming endpointing never auto-submits — helper launched without --fast #5896).
  2. Streaming transcripts now finalize mid-stream (frequent finals) — declared; this is the fix's mechanism, and the frontend already accumulates multiple finals.
  3. One-shot batch transcription unchanged — declared, pinned by test.
  4. Three tests pin both sides of the Python↔Swift argv contract — justified: macOS-only path with no CI runner, so the argv is the only testable contract.

No new config key, Python flag, or public surface; every item is declared in the description.

Subtractions

  • case "--fast": in StreamTranscribe.swift is now a knob every caller turns on: the helper has exactly 1 production spawn site (grep stream_helper_pathapple_speech/__init__.py:813), which always passes it, and the batch path uses a different source (AppleTranscribe.swift, 0 --fast hits). Insert .frequentFinalization unconditionally in the streaming helper and delete the flag, the new argv element, and test_swift_helper_still_accepts_fast — the cross-process contract that test exists to protect disappears with the flag.

[FIRST-PRINCIPLES-REVIEWED] ae9f5e9

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Aug 25, 2026
@CrysisDeu

Copy link
Copy Markdown
Collaborator Author

🤖 Kiro Crew Auto-Pipeline [operator: CrysisDeu#0c98c3a2]

Status: functionally review-ready. Everything this PR owns is green; the remaining reds are inherited from main and cannot be fixed from this branch.

No action needed on this branch until the unblockers merge; a rebase + re-run then makes the board fully green.

@bolichen97
bolichen97 enabled auto-merge August 30, 2026 00:01
@iamwhatever iamwhatever added the needs-pr-triage PR scanner: awaiting automated triage label Aug 30, 2026
@NicholasRBowers NicholasRBowers added drive-to-green PR claimed by drive-to-green pipeline and removed needs-pr-triage PR scanner: awaiting automated triage labels Aug 30, 2026
@NicholasRBowers

Copy link
Copy Markdown
Contributor

🤖 Kiro Crew [operator: NicholasRBowers#a942f9ca]: This PR has been inactive for 7+ days with failing CI. I've assessed the blockers and they appear resolvable — I'll push fixes directly to this branch as a co-author.

Assessment: All owned checks and all 5 review lanes are green; the only reds (Backend Tests shards 2 & 4 + Coverage Gate cascade) are inherited main breakages (#5879, #5846) that are now fixed on main — a rebase onto current green main + re-push clears them. No code changes expected.

If you'd prefer I don't touch this PR, add the pr-no-autofix label.

The streaming argv omitted --fast, so the DictationTranscriber ran
without .frequentFinalization and emitted only volatile partials while
the audio stream stayed open. The dashboard's semantic endpointer
schedules its utterance-complete judgment exclusively from note_final(),
so with zero mid-stream finals "Auto-submit when I finish speaking"
could never fire and recording stayed active until manually stopped.

Adding --fast makes the helper emit mid-stream finals that feed the
endpointer. The one-shot batch path keeps omitting the flag: its final
arrives when the stream closes, and frequent finalization can trade
accuracy for latency it has no use for. The live helper is macOS-only
with no CI runner coverage, so argv-pinning tests lock the flag in on
the streaming path and pin its absence on the batch path.

Closes #5896

Original work by Zezhen Xu (CrysisDeu). Rebased onto current main by Kiro Crew.

Co-authored-by: Kiro Crew <noreply@kirodotdev.github.io>
@NicholasRBowers
NicholasRBowers force-pushed the fix/apple-speech-streaming-fast-5896 branch from d8abad0 to ae9f5e9 Compare August 30, 2026 19:09
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Aug 30, 2026
@NicholasRBowers

Copy link
Copy Markdown
Contributor

🤖 Kiro Crew [operator: NicholasRBowers#a942f9ca]

Rebase-only push: d8abad090ae9f5e94f

The only red checks on this PR (Backend Tests shards 2 & 4 on 3.10/3.12/Windows, plus the downstream Coverage Gate / PR Readiness cascade) were inherited from two main-side breakages — #5879 (test_kiro_usage_api.py::TestWindowsCliStore) and #5846 (test_trust_reads.py::TestIsReadOnlyBash) — both verified to reproduce on unmodified main and both now fixed upstream. Main's last four completed ci.yml runs are green.

What this push changes:

  • Rebased the branch onto current green origin/main (was 679 commits behind). Zero conflicts; the diff is byte-identical in scope — only src/kiro_crew/apple_speech/__init__.py and test/test_apple_speech.py, exactly as before.
  • No source changes. A fresh push rebuilds refs/pull/5907/merge and re-dispatches CI cleanly (a bare gh run rerun --failed would reuse the stale pinned merge commit and re-fail).
  • Commit identity hygiene: the author field carried a non-public internal email; amended to the author's GitHub noreply identity (Zezhen Xu <32421101+CrysisDeu@users.noreply.github.com>) with name credit preserved in the commit body, plus a Co-authored-by: Kiro Crew trailer for the rebase mechanics.

Local gates before pushing (all green): isort, flake8, mypy (1177 files, no issues), black on the two changed files, and pytest test/test_apple_speech.py (68 passed, 3 skipped).

All five AI review lanes passed on the identical diff at d8abad090; they will re-run on this head. Monitoring CI to green.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Aug 30, 2026
@NicholasRBowers

Copy link
Copy Markdown
Contributor

🤖 Kiro Crew [operator: NicholasRBowers#a942f9ca]

Flake triage — re-ran Backend Tests (3.12, 1), no code change

The round's single root failure was test/test_cli_logging.py::TestDrainBeforeHardExit::test_no_listener_is_a_silent_no_op (asserts a QueueListener is None; 1 failed / 18,918 passed on the shard). Coverage Gate and PR Readiness reds are downstream cascade of that one shard.

Evidence it is cross-test global-state pollution, not this PR's diff:

  • The file is untouched by this PR (diff is apple_speech/__init__.py + test/test_apple_speech.py only).
  • The full test_cli_logging.py file passes locally on this exact head (33 passed).
  • The identical shard passed on 3.10 and on Windows in the same round.
  • The test is green on main's last four completed CI runs.

Dispatched gh run rerun --failed on run 33330084606 (same merge ref — freshly built from current main ~35 min ago, so no stale-base concern). Monitoring continues.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Aug 30, 2026
@NicholasRBowers

Copy link
Copy Markdown
Contributor

🤖 Kiro Crew [operator: NicholasRBowers#a942f9ca]

Review-ready ✅ — head ae9f5e94f, 68/68 checks green

Drive summary:

  • Rebased the branch onto current green main (was 679 commits behind); zero conflicts, no source changes — the diff remains exactly src/kiro_crew/apple_speech/__init__.py + test/test_apple_speech.py as the author wrote it.
  • Commit hygiene: author identity moved to GitHub noreply form (internal email removed), name credit in the body, Co-authored-by: Kiro Crew trailer.
  • One CI flake weathered: test_cli_logging.py::TestDrainBeforeHardExit::test_no_listener_is_a_silent_no_op failed once on Backend Tests (3.12, 1) — cross-test QueueListener pollution in a file this PR doesn't touch (passed locally, on 3.10, on Windows, and on main). Cleared on a single job re-run with no code change.
  • All five AI review lanes PASS on this head (GPT 5.6, Opus 4.8, Design, UX, First Principles), PR Readiness passed, MERGEABLE, 0 unresolved threads. Closes Apple Speech streaming endpointing never auto-submits — helper launched without --fast #5896 on merge.

Auto-merge is deliberately NOT armed — merge decision is left to maintainers. Thanks to Zezhen Xu (@CrysisDeu) for the original fix.

@NicholasRBowers NicholasRBowers left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (2 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: fix with clear root cause -- streaming helper argv gains --fast so mid-stream FINAL segments reach the endpointer's note_final(), enabling auto-submit (#5896); argv contract pinned by new tests on both streaming and batch paths.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Readiness passed on the current head, all checks green, mergeable against main, and no outstanding change requests from any reviewer. Approving as part of a maintainer sweep of fix-type PRs.

@bolichen97
bolichen97 merged commit 3d51533 into main Aug 30, 2026
106 of 108 checks passed
@bolichen97
bolichen97 deleted the fix/apple-speech-streaming-fast-5896 branch August 30, 2026 21:03

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (2 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: fix streaming speech auto-submit by passing --fast to the streaming helper argv (batch path deliberately excluded), pinned by argv + Swift-source tests.

@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Aug 30, 2026
@bolichen97 bolichen97 removed the drive-to-green PR claimed by drive-to-green pipeline label Aug 30, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (2 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: fix with a clear root cause — the default DictationTranscriber emits only volatile partials so the endpointer's note_final()-driven auto-submit never fires; adds the static --fast switch to the already-sandboxed streaming-helper argv to enable .frequentFinalization (#5896).

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (2 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: clear root cause for #5896 -- the streaming helper was spawned without --fast, so the transcriber never inserted .frequentFinalization and emitted only volatile partials until the stream closed; the dashboard endpointer schedules its utterance-complete judgment solely from note_final(), so auto-submit could never fire. The production change is one bare switch added to the existing sandboxed argv; the sandbox wrapper itself is untouched and the one-shot batch path deliberately keeps the flag off.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (2 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: clear root cause (#5896) -- the streaming speech helper omitted --fast so no mid-stream finals were emitted and auto-submit could never fire; flag added with argv-pin tests on both paths.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Apple Speech streaming endpointing never auto-submits — helper launched without --fast

4 participants