Skip to content

Add WHISPER_COMPUTE_TYPE and other CPU optomizations#3

Open
toddejohnson wants to merge 1 commit into
Thinline-Dynamic-Solutions:mainfrom
toddejohnson:cpu
Open

Add WHISPER_COMPUTE_TYPE and other CPU optomizations#3
toddejohnson wants to merge 1 commit into
Thinline-Dynamic-Solutions:mainfrom
toddejohnson:cpu

Conversation

@toddejohnson

Copy link
Copy Markdown
Contributor

Add WHISPER_COMPUTE_TYPE env var so it can be forced. Also auto detected CPU and default to int8 to work around the below error:

/app/whisper_server.py:35: DeprecationWarning:
        on_event is deprecated, use lifespan event handlers instead.

        Read more about it in the
        [FastAPI docs for Lifespan Events](https://fastapi.tiangolo.com/advanced/events/).

  @app.on_event("startup")
INFO:     Started server process [7]
INFO:     Waiting for application startup.
22:41:08  INFO  Loading Whisper model 'large-v3' on device 'cpu' (pid 7)...
22:41:09  INFO  HTTP Request: GET https://huggingface.co/api/models/Systran/faster-whisper-large-v3/revision/main "HTTP/1.1 200 OK"
ERROR:    Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 638, in lifespan
    async with self.lifespan_context(app) as maybe_state:
  File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 241, in __aenter__
    await self._router._startup()
  File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 4884, in _startup
    await handler()
  File "/app/whisper_server.py", line 48, in load_whisper_model
    model = WhisperModel(
  File "/usr/local/lib/python3.10/dist-packages/faster_whisper/transcribe.py", line 689, in __init__
    self.model = ctranslate2.models.Whisper(
ValueError: Requested float16 compute type, but the target device or backend do not support efficient float16 computation.

ERROR:    Application startup failed. Exiting.
Starting Whisper server: model=large-v3 device=cpu workers=1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant