fix(asr): unload idle speech model to release wired memory (#548)#619
fix(asr): unload idle speech model to release wired memory (#548)#619devzahirul wants to merge 3 commits into
Conversation
…#548) Loaded speech models were never released: ASRService caches every provider and only drops them on quit or model switch, so a multi-GB model stayed resident from the startup auto-load until app exit, pinning wired memory and freezing 8 GB Macs. Add an idle unload that mirrors the audio-engine standby pattern: - release all cached in-process providers after a configurable window without dictation (default 30 min; Never/5/15/30/60 in Voice Engine settings); disk caches stay, and ensureAsrReady() reloads the model on next use exactly like the first dictation after launch - guard the unload with a model-use token held across stop()'s final transcription, Local API requests, and file transcription jobs, and re-check recording/preparation/download state at fire time - skip Apple Speech engines, which hold no in-process weights - clear nemotronProviders in resetTranscriptionProvider(), which leaked loaded Nemotron models when switching away from them
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: edd0454893
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| self.externalCoreMLProvider = nil | ||
| self.nemotronProviders.removeAll() | ||
| self.whisperProvider = nil | ||
| self.isAsrReady = false |
There was a problem hiding this comment.
When the idle timer fires, this leaves isAsrReady false; the next start() call does not call ensureAsrReady() before startStreamingTranscription(), and that helper immediately returns while isAsrReady is false. For streaming-capable models, the entire first dictation after an idle timeout loses live partial transcription and only reloads in stop(), so the model should be reloaded before starting the streaming loop if the intent is to reload on the next dictation.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Good catch — fixed in 9063f86. start() now branches: if the model is loaded it starts streaming immediately as before; otherwise it defers via attachStreamingTranscriptionWhenModelLoads(), which awaits ensureAsrReady() (single-flighted with the existing ContentView pre-load task) and attaches the streaming loop to the same session once ready, guarded by the session ID, isRunning, and a re-check of supportsStreaming. This also improves the pre-existing cold-launch case where recording started before the startup auto-load finished.
|
Thanks for PR. Why is it under voice engine? I thought this was for Fluid Intelligence and not Voice model? |
Greptile SummaryThis PR fixes a wired-memory leak in
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
…d setting UI (altic-dev#548) Address review feedback on the idle-unload PR: - start(): when capture begins with no loaded model (idle unload, or app launch before auto-load finishes), reload during capture and attach the live streaming preview to the same session once ready, instead of losing streaming for the whole dictation - hide the Unload Model When Inactive row while an Apple Speech engine is selected, since those hold no in-process model to unload - label non-standard restored values grammatically (After 1 minute)
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9063f8686e
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| "Speech model idle unload armed for \(minutes)min (\(reason))", | ||
| source: "ASRService" | ||
| ) | ||
| let delay = UInt64(minutes) * 60 * 1_000_000_000 |
There was a problem hiding this comment.
If a restored backup or UserDefaults entry contains a very large positive speechModelIdleUnloadMinutes value, it reaches this calculation because the setting getter only clamps the lower bound, and the chained UInt64 multiplications can overflow and trap when the idle timer is scheduled after model load or a policy change. Clamp to a sane maximum or use overflow-safe duration construction before arming the task.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in fd0821f — reads and writes are now clamped to 0…1440 minutes (clampedSpeechModelIdleUnloadMinutes), so the nanosecond multiplication is bounded well inside UInt64 range, and the unit test covers Int.max both via the setter and written directly to defaults.
…erflow-safe (altic-dev#548) A hand-edited or corrupt restored value could previously reach the UInt64 nanosecond multiplication unbounded and trap when the unload timer was armed. Clamp reads and writes to 0...1440 minutes.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fd0821f105
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| self.externalCoreMLProvider = nil | ||
| self.nemotronProviders.removeAll() | ||
| self.whisperProvider = nil | ||
| self.isAsrReady = false |
There was a problem hiding this comment.
When the idle timer unloads a model, this flips isAsrReady to false while leaving modelsExistOnDisk true, but the in-app RecordingView still disables its Start Recording button whenever !asr.isAsrReady && !asr.isRunning (Sources/Fluid/UI/RecordingView.swift:77). After the default idle window, users who start dictation from that UI can no longer trigger the intended transparent reload; update the UI gating/readiness state to allow cached idle-unloaded models to start.
Useful? React with 👍 / 👎.
|
Good question — you're right, and I've updated the PR description to be precise about it. Two separate things are going on in #548:
To fully resolve the reporter's case, the same idle policy needs to apply to the Fluid Intelligence runtime — likely armed on the same "minutes since last use" clock but tracked in |
2 participants
Description
Fixes the voice-engine side of the wired-memory pinning reported in #548: once a speech model is loaded, FluidVoice never releases it. (The reporter's specific 3.43 GB
fluid-1-q4_k_m.ggufis the Fluid Intelligence runtime — same missing-idle-unload defect, separate code path; see the Related Issue section and PR discussion.)ASRServicecaches every provider instance (FluidAudioProvideralone holds two to threeAsrManagerCoreML models,WhisperProvidera Metal-backed GGUF session) and only drops them on app quit or model switch — there is no time-based offload anywhere in the codebase. Combined with the startup auto-load ininitialize(), a multi-GB model stays resident from launch until quit, which matches the reporter's measurements (wired stayed at ~5.6–5.8 GB for 10+ idle hours and only returned to baseline on quit) and freezes 8 GB Macs.This PR adds an idle unload for the speech model, mirroring the existing audio-engine standby pattern:
ASRServicedrops all cached in-process providers and frees the model memory. Disk caches are untouched.startRecording()already pre-loads the model in the background), whilestop()awaitsensureAsrReady()— exactly the same path as the first dictation after app launch.beginSpeechModelUse()/endSpeechModelUse()) guarantees the unload can never fire while a final transcription, a Local API request, or a long file-transcription job is in flight; recording, preparation, and download states are re-checked at fire time via the pure decision functioncanUnloadIdleSpeechModel(...).holdsModelInProcessMemory == false) — they run in system services and hold no in-process weights, so there is nothing to unload.parakeetVocabularyDidChange), and the value participates in settings backup/restore.Also fixes a latent leak in the same lifecycle:
resetTranscriptionProvider()cleared every cached provider exceptnemotronProviders, so switching away from a Nemotron model kept its CoreML models in memory forever (shutdownForTermination()already removed them; the reset path now matches).Verified end-to-end
Release build,
whisper-tiny, unload window set to 1 minute:Process RSS dropped from 236.7 MB to 160.6 MB the moment the unload fired (whisper-tiny is only a 46 MB GGUF; with the ~3.4 GB model from the report this is multi-GB of wired memory). The app stays fully functional afterwards — the next dictation reloads through the existing
ensureAsrReady()flow.Type of Change
Related Issue or Discussion
Part of #548 — this PR fixes the voice-engine side of the leak (ASR providers are never released while the app runs, and
resetTranscriptionProvider()additionally leaksnemotronProviderson model switch). The reporter's specific 3.43 GB file (fluid-1-q4_k_m.gguf) is the Fluid Intelligence runtime, which has the same missing idle-unload and needs the same policy applied — happy to extend this PR or send it as a follow-up, whichever is preferred (see PR discussion).Testing
swiftlint --strict --config .swiftlint.yml Sources— 0 violationsswiftformat --config .swiftformat Sourcesxcodebuild testsuite (same invocation as CI) — all green, including the newSpeechModelIdleUnloadTests(7 tests covering the unload decision truth table, the in-process-memory model classification, and the setting's default/clamping)Screenshots / Video
New "Unload Model When Inactive" row in Settings → Voice Engine (below Remove Filler Words):
Full Voice Engine card showing placement (below Remove Filler Words)
Notes
stop()for the reload, identical to the first dictation after app launch. For most models this is a few seconds; the background pre-load during recording absorbs it entirely for normal-length dictations.