feat: add deep health and readiness checks (AURA-RM-002)

[brandon-shelton-mezmo]

Summary

• Add /health/live (liveness) and /health/ready (readiness) endpoints with per-subsystem connectivity verification for LLM providers, MCP servers, and vector stores
• Lightweight probes: HTTP model-listing for LLM, GET for MCP, REST health for Qdrant, AWS credential resolution for Bedrock
• Cached results with configurable TTL (HEALTH_CHECK_CACHE_TTL_SECS, default 10s)
• Per-probe timeout (HEALTH_CHECK_TIMEOUT_SECS, default 5s)
• Prometheus metrics: aura_health_ready gauge, per-subsystem aura_health_subsystem_status gauges, aura_health_check_duration_seconds histogram
• Sanitized error messages in health responses (no internal IPs/hostnames)
• Shutdown guard exemption for /health/* paths
• Existing /health endpoint unchanged (backward compatible)

Note: Includes AURA-RM-008 (error taxonomy + prometheus metrics) commits as prerequisites. Those are also in PR #80.

Test plan

• cargo fmt --check passes
• cargo clippy --workspace --all-targets -- -D warnings passes
• cargo test --workspace --lib — 48 tests pass (includes 17 new health check tests)
• Verify /health/live returns 200 {"status": "alive"}
• Verify /health/ready returns 200 with per-subsystem JSON when healthy, 503 when unhealthy
• Verify existing /health endpoint unchanged
• Verify Docker Compose healthcheck still works
• Jenkins integration tests pass

🤖 Generated with Claude Code

[promptless]

Promptless prepared a documentation update related to this change.

Triggered by PR #117

Added docs/observability.md documenting the new /health/live and /health/ready endpoints, environment variable configuration, and Prometheus health metrics.

Review: Add observability docs

[kristof-mattei]

Unneeded cast to u64, we cast to f64 below.

[kristof-mattei]

We can avoid cloning here to save an allocation:

    let tool_name_for_metrics = state
        .tool_call_map
        .get(&tool_result.id)
        .map(|(name, _)| &**name)
        .unwrap_or_else(|| "unknown");

[kristof-mattei]

Why not take in an std::time::Duration and convert internally to what you want to display? That way we can leverage the type system to avoid someone misreading the parameter name and send you seconds or milliseconds or nanoseconds.

[kristof-mattei]

.len() on a string returns the amount of bytes, which is different from the character length (i.e. the width on screen) you're testing here. And even that isn't 100% if we consider graphemes.

It probably doesn't matter in this code base, where we only have tools with ASCII names, but ideally we would do something like tool.chars().count() to check the UTF-8 character count.

But wouldn't it be better to truncate?

[kristof-mattei]

This requires some more thought.

When a lock is poisoned it means that somewhere in the codebase a lock was acquired and panicked before the lock was properly dropped.

Which means that this moment we cannot make any assumptions of the contents of the lock.

1. While the std library does it's very best to ensure that even in the case of a panic the internal state of the HashMap remains consistent, it's worth thinking about whether we want to rely on that mechanism, and we probably should document our thinking.

2. Equally if there are multiple code-paths accessing KNOWN_TOOLS, that do multiple operations while holding a single lock, we don't know whether these changes have been atomically committed. If we do a remove and an add within the same lock, it might be that the remove was called, but the add was not.

While the HashMap is still in a valid state from Rust's point of view, it now violates the invariants we have placed on top of it.

For 1, this is something we don't really control. The only way I can see corruption there is when the thread would forcefully get shot down, which... shouldn't happen.

For 2, If we are able to validate that the usages of KNOWN_TOOLS, and the taking of the lock are atomically, or we don't care about the overall state of the HashMap itself, we can do into_inner(). However, in the case that it is poisoned (which we know based on the lock() result we probably want to clear the the poison with clear_poison.

On the other hand, if we're not able (or don't wish) to continue with a state we can't trust, we can always re-init the HashMap, losing our state.

In either case, I'd love to see an ERROR or WARN log when the lock is poisoned, because when that happens, we probably have an issue somewhere else.

[kristof-mattei]

We probably don't need the clone here.

[kristof-mattei]

.to_owned() to get an owned instance of a &str.

[kristof-mattei]

These should be moved to the top.

[kristof-mattei]

We have an opportunity here to have health_check_timeout as Duration, then internally we don't have to convert it. We can still name env variable as ..._SECS and convert it internally.

Same for the long option.

This way we use the type system to pass on a Duration, and no-one internally needs to worry about whether it's seconds, ...

A u64 has no provenance of whether it came from seconds or milliseconds or ...

Story about a couple of dollars lost because 2 teams used different units and stored them in a type that didn't carry that information:

An investigation attributed the failure to a measurement mismatch between two measurement systems: SI units (metric) by NASA and US customary units by spacecraft builder Lockheed Martin.

https://en.wikipedia.org/wiki/Mars_Climate_Orbiter

[kristof-mattei]

General remark: many places where to_string() should be replaced with to_owned().

[kristof-mattei]

#[arg(
    env = "HEALTH_CHECK_TIMEOUT_SECS",
    default_value = "5",
    long = "health_check-timeout-secs",
    help = "...",
    value_parser = parse_duration_from_secs
)]
pub health_check_timeout: Duration,
...


fn parse_duration_from_secs(value: &str) -> Result<Duration, String> {
    let seconds = value
        .parse()
        .map_err(|error| format!("Could not parse `{}`: {}", value, error))?;

    Ok(Duration::from_secs(seconds))
}

Using this you leverage clap to do the conversion for you, and no-where in accessible code do we have access to the broadly typed version.