Skip to content

fix(#114): open the listener before cache warmup, add /health and /health/live - #115

Merged
jrosskopf merged 4 commits into
mainfrom
fix/114-readiness-during-warmup
Sep 13, 2026
Merged

jrosskopf merged 4 commits into
mainfrom
fix/114-readiness-during-warmup

Conversation

@jrosskopf

@jrosskopf jrosskopf commented Sep 13, 2026

Copy link
Copy Markdown
Contributor

Problem

Cache warmup runs synchronously before the HTTP listener opens, so a deployment whose cache takes
longer than the platform's maximum health-check window can never go live: every attempt fails its
health check and is rolled back. There is also no general health endpoint — /api/v1/_config/health
requires --config-service and /mcp/health is MCP-only — which forces operators to point health
checks at a data endpoint, precisely the thing that cannot answer during warmup.

Verified in source on main:

main()
  └─ initializeDatabase()
       └─ DatabaseManager::initializeDBManagerFromConfig()   database_manager.cpp:116  lock_guard(db_mutex)
            └─ cache_manager->warmUpCaches(...)              database_manager.cpp:170  ← still under the lock
  ...
  └─ std::thread unified_server_thread                       ← the socket opens only here

Closes #114.

What changed

Startup order. The listener opens first; warmup moves to a background thread. This also takes
warmUpCaches() out from under db_mutex — the lock's scope is unchanged, but a warmup worker
can no longer deadlock against it via the VFS provider on a remote template.path.

Health endpoints, both always registered, no auth, no config service:

Route Behaviour
GET /health/live liveness — 200 as soon as the process is up. Point platform health checks here.
GET /health readiness — 503 starting200 ready, or 503 degraded naming failed caches

The 503 contract. An endpoint whose cache is not ready returns 503 + Retry-After, never a
partial or empty result — serving a half-built cache would be a worse bug than the one being fixed.
The gate sits in handleRequest before the method switch, so GET, writes and DELETE are all
covered, and is also called from the MCP tool and MCP resource paths.

Failures are contained. A cache that fails to build no longer takes the process down; /health
reports degraded with the detail, and the endpoint returns 503 rather than 200-with-nothing.

Concurrency the early socket introduces. HeartbeatWorker already calls refreshCache() on its
own thread, and now runs while warmup is still going. An in-flight registry keyed by
(catalog, schema, table) makes a duplicate concurrent refresh a logged no-op, and refreshCache()
owns terminal state so warmup, heartbeat and the config service all converge — previously a
heartbeat that won the race and then threw could pin a cache at starting for the process lifetime.

Config service refresh/GC used a throwaway CacheManager with its own empty registries, so a
manual refresh could collide with warmup and could never clear a failed cache. It now uses the
shared instance.

Test fixtures. wait_for_server_healthy accepted any HTTP status from / as "ready" — a valid
proxy only while the socket opened after warmup. It now polls GET /health for 200, as does the
integration-test-ci target, which was a bare sleep 5.

Testing

C++ unit 684/684 pass (was 672; 12 added)
Integration, full suite, cold DuckLake 570 passed, 26 skipped
New test_warmup_readiness.py, patched binary 5/5 pass
New test_warmup_readiness.py, pre-fix binary 5/5 fail

The last two rows are the point: the new tests fail against a binary built from main and pass
against this branch, so they genuinely capture the defect rather than passing vacuously.

The five integration tests run a real binary over real HTTP against a slow-cache fixture and assert:
liveness answers during warmup; readiness and the cached endpoint are unavailable until warmup
finishes; the endpoint serves correct rows afterwards; a failed cache keeps the process alive with
the endpoint unavailable; and a scheduled refresh firing during warmup is suppressed.

Pre-existing failures, unrelated to this PR: test/integration/test_https_config.py fails 7/11.
Its fixture returns a relative binary path and then runs subprocess with a different cwd. A
clean main worktree using the same binary produces an identical failure set. Worth a separate
issue.

Upgrade note

Behaviour change: the socket now opens before caches are populated. Anything relying on
"connection refused == not ready" should use GET /health instead, which preserves those
semantics. For AWS App Runner and similar, point the health check at GET /health/live.

Deliberately out of scope

  • Parallel cache warmup. It would not fix this issue — the reporting deployment has a single
    slow cache, and parallelism across caches cannot help that. Speedup is bounded by
    sum(tᵢ)/max(tᵢ).
  • Narrowing db_mutex to the db handle lifecycle.
  • Seeding readiness as ready for cache tables that already exist, so a restart with a complete
    persisted snapshot doesn't reopen a 503 window. Worth a follow-up.
  • Cancellable warmupSIGTERM during a long warmup still waits for the in-flight query.
  • Sanitising /health failure detail; it currently echoes the underlying exception text.

Design notes: docs/plans/114-readiness-during-warmup.md.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

jrosskopf and others added 4 commits September 13, 2026 14:59
Open the HTTP listener before cache warmup and run warmup on a background
thread, so a slow cache can no longer make a deployment fail its platform
health check.

- add /health/live (liveness, always 200) and /health (readiness, 503 until
  every cache is built); previously the only health routes were behind
  --config-service or MCP-only
- track cache readiness per (catalog, schema, table); cached endpoints return
  503 + Retry-After until ready, never a partial or empty result
- contain warmup failures: the process stays up and /health reports degraded
- suppress duplicate in-flight refreshes for the same table, so the heartbeat
  worker cannot collide with warmup now that the socket opens earlier

warmUpCaches() moves out of initializeDBManagerFromConfig(), which also takes
it out from under db_mutex. The lock's scope is unchanged.

Closes #114
Review follow-ups:

- hoist the readiness check out of handleGetRequest into handleRequest
  before the method switch, so writes and DELETE are gated too, and call
  it from the MCP tool and MCP resource paths. Previously only REST GET
  was gated, so an MCP call or a write could run against an unbuilt cache
  - and a write committed against the old table was silently destroyed
  by warmup's CREATE OR REPLACE.
- config service refresh/GC used a throwaway CacheManager, bypassing both
  the in-flight registry and the readiness map; use the shared instance so
  a manual refresh can recover a failed cache without a restart.
- refreshCache now marks terminal failure before rethrowing, and warmUpCaches
  waits while a duplicate refresh still owns Starting - otherwise a heartbeat
  that won the race and threw pinned /health to starting for the process
  lifetime.
- integration fixtures poll GET /health instead of treating any HTTP status
  as ready; that proxy was only valid while the socket opened after warmup.
…onse ODR violation

The two readiness-gate tests added in the previous commit passed locally but
failed on the CI image (Ubuntu 24.04 / GCC 13) with 200 instead of 503.

Cause: CROW_ENABLE_COMPRESSION was defined by api_server.hpp, request_handler.hpp
and mcp_route_handlers.hpp rather than by the build. The macro adds a `compressed`
member to crow::response, so the class layout depends on whether a translation
unit reached <crow.h> before or after one of those headers. CMakeLists.txt set it
only as a CMake variable, which never becomes a -D.

Reproduced in the CI container: the same crow::response object, at the same
address with byte-identical storage, reported is_completed() == 0 in the test TU
and a garbage non-zero value in request_handler.cpp — the two TUs read
`completed_` from different offsets. handleRequest therefore took its
"response already completed" early return and never reached the readiness gate.

Fixes:
- define CROW_ENABLE_COMPRESSION via add_compile_definitions, drop the three
  per-header defines, so every TU shares one layout;
- include <crow.h> before the `#define private public` block in the four test
  files that use it, so crow is never parsed with rewritten access specifiers
  (which independently corrupted the layout in those TUs).

Verified in the CI docker image: 684/684 pass. Also 684/684 locally.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@jrosskopf
jrosskopf merged commit 51c5d80 into main Sep 13, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cache warmup blocks HTTP server startup (undeployable on platforms with capped health-check windows (e.g. AWS App Runner))

1 participant