[Prudctual]

Summary

Streaming OpenAI-compatible clients that send a request no worker can serve currently get HTTP 200 and an SSE frame with a sanitized "Internal server error". The same request with stream: false already returns HTTP 400 with the worker's reason.

The peek-before-200 path already exists (check_for_backend_error + pre_commit_error_peek_timeout()). It was off by default (DYN_HTTP_PRE_COMMIT_ERROR_PEEK_MS unset → skip peek). This PR takes Option A: default a 100ms peek window when the env var is unset.

That is the better of the two options in ai-dynamo#14748:

• The peek returns as soon as the first non-annotation event arrives, so a healthy stream pays min(time-to-first-event, 100ms), not a fixed delay. Local happy-path streaming still commits HTTP 200 with tokens at ~0ms after the first chunk.
• It covers every backend and every first-frame guard (Python generator __anext__ capability checks, Backend(InvalidArgument), etc.) without hoisting checks out of each handler.
• Option B (typed prologue / hoist capability checks) would only help guards that can run from the request + static config, and SGLang still has no equivalent call-time seam.

Set DYN_HTTP_PRE_COMMIT_ERROR_PEEK_MS=0 to restore immediate HTTP 200 commit. Unparsable values use the 100ms default.

/v1/completions streaming still has no peek; that is out of scope for this issue.

Details

• Default pre_commit_error_peek_timeout() to 100ms when DYN_HTTP_PRE_COMMIT_ERROR_PEEK_MS is unset.
• 0 still disables the peek (commit 200 immediately).
• Document the default in environment_names.rs and the frontend configuration reference.
• Tests: streaming and unary first-frame Backend(InvalidArgument) → HTTP 400 JSON with the worker message; happy-path streaming → HTTP 200 with tokens.

Where should the reviewer start?

• lib/llm/src/http/service/openai.rs — DEFAULT_PRE_COMMIT_ERROR_PEEK_MS and pre_commit_error_peek_timeout()
• lib/llm/tests/http-service.rs — streaming vs unary 4xx contract and happy-path 200

Related Issues

This PR is linked to an issue:

• Fixes Streaming clients get HTTP 200 + a sanitized 500 frame for a request no worker can serve ai-dynamo/dynamo#14748

Validation

# Unit: default 100ms, 0 disables, custom window, unparsable → default,
# and first-token peek returns before the full window.
cargo test -p dynamo-llm --lib pre_commit -- --test-threads=1

Result: ok. 5 passed; 0 failed (pre_commit_error_peek_timeout_*, test_pre_commit_peek_returns_on_first_token_without_waiting_full_window).

cargo test -p dynamo-llm --test http-service -- --test-threads=1 \
  4xx_on_backend_invalid_argument \
  happy_path_still_200 \
  test_http_service \
  test_sse_keep_alive \
  test_client_disconnect_cancellation_streaming

Result: ok. 6 passed; 0 failed, including:

• test_chat_completions_returns_4xx_on_backend_invalid_argument — streaming and unary first-frame Backend(InvalidArgument) → HTTP 400 JSON with the worker message (not SSE 500).
• test_streaming_responses_returns_4xx_on_backend_invalid_argument — same contract on /v1/responses with the env var unset.
• test_streaming_chat_completions_happy_path_still_200_with_tokens — HTTP 200, text/event-stream, generated tokens, no sanitized error frame. Frontend log: status=200 latency_ms=0.
• test_http_service, test_sse_keep_alive_emits_comments_during_idle_stream, test_client_disconnect_cancellation_streaming — existing streaming coverage still passes (idle first token waits the 100ms cap, then 200).

cargo clippy --locked --no-deps -p dynamo-llm -p dynamo-runtime --all-targets -- -D warnings
python3 docs/fern/scripts/docs_lint.py --scan docs

Clippy: clean. Docs lint: 0 errors.

AI assistance

This change was implemented with assistance from Cursor Grok 4.6. I reviewed the streaming vs unary error path, chose Option A over hoisting capability checks, and own the code quality of the diff.

Fixes ai-dynamo#14748

---

Upstream target: this fork PR is not the merge target. Open the contribution against ai-dynamo/dynamo:main from:

https://github.com/ai-dynamo/dynamo/compare/main...Prudctual:cursor/frontend-pre-stream-refusals-974c?expand=1

Automation cannot create that PR: POST /repos/ai-dynamo/dynamo/pulls returns 403 Resource not accessible by personal access token.

  

---

Summary by cubic

Fixes ai-dynamo#14748. Streaming /v1/chat/completions and /v1/responses requests a backend refuses now return HTTP 4xx with the worker's error instead of HTTP 200 plus a sanitized SSE "Internal server error" frame. The default peek window now lives in BackendErrorCheck::parse and stays on by default at 100ms; it returns as soon as the first real event arrives, so a healthy stream pays min(time-to-first-event, 100ms).

• Set DYN_HTTP_PRE_COMMIT_ERROR_PEEK_MS=0 to disable the peek and commit HTTP 200 immediately; unparsable values warn and keep the 100ms default.
• Repeated request_id annotations no longer fill the pre-commit buffer and skip the peek; at most one frame is emitted.
• Adds streaming 4xx contract coverage plus happy-path 200 tests on both endpoints, ported to BackendErrorCheck after the main merge replaced pre_commit_error_peek_timeout() and removed the unary arm.

^{Written for commit 35d828f. Summary will update on new commits.}