fix(#114): open the listener before cache warmup, add /health and /health/live - #115
Merged
Merged
Conversation
Open the HTTP listener before cache warmup and run warmup on a background thread, so a slow cache can no longer make a deployment fail its platform health check. - add /health/live (liveness, always 200) and /health (readiness, 503 until every cache is built); previously the only health routes were behind --config-service or MCP-only - track cache readiness per (catalog, schema, table); cached endpoints return 503 + Retry-After until ready, never a partial or empty result - contain warmup failures: the process stays up and /health reports degraded - suppress duplicate in-flight refreshes for the same table, so the heartbeat worker cannot collide with warmup now that the socket opens earlier warmUpCaches() moves out of initializeDBManagerFromConfig(), which also takes it out from under db_mutex. The lock's scope is unchanged. Closes #114
Review follow-ups: - hoist the readiness check out of handleGetRequest into handleRequest before the method switch, so writes and DELETE are gated too, and call it from the MCP tool and MCP resource paths. Previously only REST GET was gated, so an MCP call or a write could run against an unbuilt cache - and a write committed against the old table was silently destroyed by warmup's CREATE OR REPLACE. - config service refresh/GC used a throwaway CacheManager, bypassing both the in-flight registry and the readiness map; use the shared instance so a manual refresh can recover a failed cache without a restart. - refreshCache now marks terminal failure before rethrowing, and warmUpCaches waits while a duplicate refresh still owns Starting - otherwise a heartbeat that won the race and threw pinned /health to starting for the process lifetime. - integration fixtures poll GET /health instead of treating any HTTP status as ready; that proxy was only valid while the socket opened after warmup.
…onse ODR violation The two readiness-gate tests added in the previous commit passed locally but failed on the CI image (Ubuntu 24.04 / GCC 13) with 200 instead of 503. Cause: CROW_ENABLE_COMPRESSION was defined by api_server.hpp, request_handler.hpp and mcp_route_handlers.hpp rather than by the build. The macro adds a `compressed` member to crow::response, so the class layout depends on whether a translation unit reached <crow.h> before or after one of those headers. CMakeLists.txt set it only as a CMake variable, which never becomes a -D. Reproduced in the CI container: the same crow::response object, at the same address with byte-identical storage, reported is_completed() == 0 in the test TU and a garbage non-zero value in request_handler.cpp — the two TUs read `completed_` from different offsets. handleRequest therefore took its "response already completed" early return and never reached the readiness gate. Fixes: - define CROW_ENABLE_COMPRESSION via add_compile_definitions, drop the three per-header defines, so every TU shares one layout; - include <crow.h> before the `#define private public` block in the four test files that use it, so crow is never parsed with rewritten access specifiers (which independently corrupted the layout in those TUs). Verified in the CI docker image: 684/684 pass. Also 684/684 locally. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Cache warmup runs synchronously before the HTTP listener opens, so a deployment whose cache takes
longer than the platform's maximum health-check window can never go live: every attempt fails its
health check and is rolled back. There is also no general health endpoint —
/api/v1/_config/healthrequires
--config-serviceand/mcp/healthis MCP-only — which forces operators to point healthchecks at a data endpoint, precisely the thing that cannot answer during warmup.
Verified in source on
main:Closes #114.
What changed
Startup order. The listener opens first; warmup moves to a background thread. This also takes
warmUpCaches()out from underdb_mutex— the lock's scope is unchanged, but a warmup workercan no longer deadlock against it via the VFS provider on a remote
template.path.Health endpoints, both always registered, no auth, no config service:
GET /health/live200as soon as the process is up. Point platform health checks here.GET /health503 starting→200 ready, or503 degradednaming failed cachesThe 503 contract. An endpoint whose cache is not ready returns
503+Retry-After, never apartial or empty result — serving a half-built cache would be a worse bug than the one being fixed.
The gate sits in
handleRequestbefore the method switch, so GET, writes and DELETE are allcovered, and is also called from the MCP tool and MCP resource paths.
Failures are contained. A cache that fails to build no longer takes the process down;
/healthreports
degradedwith the detail, and the endpoint returns 503 rather than 200-with-nothing.Concurrency the early socket introduces.
HeartbeatWorkeralready callsrefreshCache()on itsown thread, and now runs while warmup is still going. An in-flight registry keyed by
(catalog, schema, table)makes a duplicate concurrent refresh a logged no-op, andrefreshCache()owns terminal state so warmup, heartbeat and the config service all converge — previously a
heartbeat that won the race and then threw could pin a cache at
startingfor the process lifetime.Config service refresh/GC used a throwaway
CacheManagerwith its own empty registries, so amanual refresh could collide with warmup and could never clear a failed cache. It now uses the
shared instance.
Test fixtures.
wait_for_server_healthyaccepted any HTTP status from/as "ready" — a validproxy only while the socket opened after warmup. It now polls
GET /healthfor 200, as does theintegration-test-citarget, which was a baresleep 5.Testing
test_warmup_readiness.py, patched binarytest_warmup_readiness.py, pre-fix binaryThe last two rows are the point: the new tests fail against a binary built from
mainand passagainst this branch, so they genuinely capture the defect rather than passing vacuously.
The five integration tests run a real binary over real HTTP against a slow-cache fixture and assert:
liveness answers during warmup; readiness and the cached endpoint are unavailable until warmup
finishes; the endpoint serves correct rows afterwards; a failed cache keeps the process alive with
the endpoint unavailable; and a scheduled refresh firing during warmup is suppressed.
Pre-existing failures, unrelated to this PR:
test/integration/test_https_config.pyfails 7/11.Its fixture returns a relative binary path and then runs
subprocesswith a differentcwd. Aclean
mainworktree using the same binary produces an identical failure set. Worth a separateissue.
Upgrade note
Behaviour change: the socket now opens before caches are populated. Anything relying on
"connection refused == not ready" should use
GET /healthinstead, which preserves thosesemantics. For AWS App Runner and similar, point the health check at
GET /health/live.Deliberately out of scope
slow cache, and parallelism across caches cannot help that. Speedup is bounded by
sum(tᵢ)/max(tᵢ).db_mutexto thedbhandle lifecycle.readyfor cache tables that already exist, so a restart with a completepersisted snapshot doesn't reopen a 503 window. Worth a follow-up.
SIGTERMduring a long warmup still waits for the in-flight query./healthfailure detail; it currently echoes the underlying exception text.Design notes:
docs/plans/114-readiness-during-warmup.md.Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.