/health endpoint returns status 200 when model is not yet loaded

[cphlipot]

Before you start

• I have read the FAQ and my problem is not answered there.
• I have read the Roadmap and this is not already planned there.
• I have searched existing issues and found no duplicate.
• I am on the latest release, or on a freshly rebuilt main when building from source.

What happened

With most other inference engines ( llama.cpp, tabbyapi, ninfer, ds4, colibri, ik_llama.cpp, and vllm) a GET request to /health will return an error status such as 404 or 503 if the model is not yet loaded and 200 only when ready, which is a behavior other tools such as llama-swap rely on a non-successful error code to deterimine the health of the engine.

FreeToken deviates from every other implementation i could find by always returning status 200 from /health even when the model is not ready, which makes it look like a bug.

Behavior can easily be seen via running curl while model is loading:

 curl -i http://127.0.0.1:1919/health
HTTP/1.1 200 OK
date: Tue, 22 Sep 2026 16:45:43 GMT
server: uvicorn
content-length: 190
content-type: application/json

{"status":"loading","phase":"weights","progress":{"done_bytes":10032828408,"total_bytes":10816852472},"model":"Qwen3.8-Flash-Next-NVFP4","instance_id":"c66344f3-b496-4dcb-a359-90c502ecb3f9"}

note: status is "loading" but status is "200 OK", where instead something like 503 would be expected instead.

I believe we should set the error code in the /health response until model is actually loaded.

How did you install FreeToken

Built from source

FreeToken version

cab110e

OS

Fedora

OS details

44

GPU and driver

RTX 5090 615.71.09

CPU and system RAM

i7-12600k DDR4

Checkpoint

nvidia/Qwen3.8-Flash-Next-NVFP4

Command

ft serve --model models/Qwen3.8-Flash-Next-NVFP4

Full log

Anything else

No response

[sOblivionsCall]

This is a workaround, not a fix. The bug described here still stands and still needs fixing upstream — the wrapper below just lets anyone blocked by it get moving in the meantime.

Confirming the report independently on v0.1.3 (release build, not source) with Qwen3.6-35B-A3B-FP8 on Linux / CUDA 13 / RTX 4090, so it isn't specific to a model or to building from main.

There is no alternative endpoint to point a proxy at

I checked every GET endpoint the server exposes while the model was loading:

endpoint	status during load
/health	200 — body {"status":"loading","phase":"expert_banks","progress":{…}}
/v1/models	200
/v1/stats	200
/v1/cache/status	200
/metrics, /ping	404 (not implemented)

POST /v1/chat/completions does correctly return 503 {"error":"model is still loading"}, and /health flips to {"status":"ok"} once ready. But since no GET returns a non-success code while loading, there's nothing to redirect a proxy's checkEndpoint to — which rules out the obvious workaround and is why something more involved is needed.

Why raising the health-check timeout doesn't help

llama-swap decides readiness purely from the HTTP status of checkEndpoint — no body matching, no retry on an upstream error. The sequence is:

1. llama-swap starts ft serve
2. /health returns 200 within ~2 seconds
3. llama-swap marks the model ready and forwards the queued request
4. the caller gets 503 model is still loading — in my case ~90 seconds before the model was usable

healthCheckTimeout never comes into play, because the check passes immediately rather than timing out. Setting checkEndpoint: none is strictly worse.

Workaround: hold the public port closed until the server is actually ready

Run ft serve on a private port, and only open the port the proxy was told to use once /health reports ok. While the public port is closed the proxy's health check gets connection-refused, which it treats as "not up yet" and retries — so the proxy's existing wait loop does the waiting, bounded by its own healthCheckTimeout. No patch to FreeToken, no patch to llama-swap.

#!/usr/bin/env bash
# ft-gated PUBLIC_PORT [ft serve args...]
set -uo pipefail
PUBLIC_PORT="${1:?usage: ft-gated PUBLIC_PORT [ft serve args...]}"; shift
INTERNAL_PORT=$((PUBLIC_PORT + 1000))   # must not collide with the proxy's port range

SRV_PID=""; FWD_PID=""
cleanup() {
    [[ -n "$FWD_PID" ]] && kill "$FWD_PID" 2>/dev/null
    [[ -n "$SRV_PID" ]] && kill "$SRV_PID" 2>/dev/null
    [[ -n "$SRV_PID" ]] && wait "$SRV_PID" 2>/dev/null   # let it release VRAM
    return 0
}
trap cleanup EXIT INT TERM

ft serve --host 127.0.0.1 --port "$INTERNAL_PORT" "$@" &
SRV_PID=$!

while true; do
    kill -0 "$SRV_PID" 2>/dev/null || { echo "ft serve exited during startup" >&2; exit 1; }
    status=$(curl -sf --max-time 5 "http://127.0.0.1:$INTERNAL_PORT/health" 2>/dev/null \
        | python3 -c 'import sys,json; print(json.load(sys.stdin).get("status",""))' 2>/dev/null)
    [[ "$status" == "ok" ]] && break
    sleep 2
done

socat "TCP-LISTEN:$PUBLIC_PORT,fork,reuseaddr,bind=127.0.0.1" "TCP:127.0.0.1:$INTERNAL_PORT" &
FWD_PID=$!
wait -n "$SRV_PID" "$FWD_PID"   # exit if either dies, so the proxy sees the model stop

llama-swap then calls the wrapper instead of the server directly:

models:
  "Qwen3.6-35B-A3B-FP8":
    cmd: |
      /path/to/ft-gated ${PORT}
      --model /path/to/Qwen3.6-35B-A3B-FP8

Verified end to end: cold start took 98s and the first request returned normally instead of 503.

Two things that will bite you if you adapt this:

• Don't exec socat. That replaces the shell and discards the trap, so ft serve is orphaned when the proxy sends SIGTERM and its VRAM is never released — which then breaks the next model load. Background the forwarder and wait -n on both PIDs.
• Bail out if the server dies during startup. Otherwise a crash-on-load is indistinguishable from slow loading and just hangs until the proxy's timeout.

Costs one extra TCP hop for all traffic including token streaming, which is negligible next to generation latency. It can be deleted outright once /health reports readiness properly.

On the actual fix

Agreed with the proposal here — gating /health on maintenance_state != "serving" would match what llama.cpp, vLLM, TGI and the others do, and is what proxies already expect. The one thing worth deciding deliberately is whether anyone is currently treating that 200 as a liveness signal rather than a readiness one, since for them this would be a breaking change. If that's a concern, a separate /ready endpoint (or /health?ready=1) would give proxies what they need without changing existing behavior. Either way the current state — 200 while the API layer is returning 503 — seems clearly wrong.