Every HTTP route that ravel-server registers. Each route lists its method,
what it accepts, what it returns, the status codes, whether a tenant
credential is required, which process modes serve it, and its cargo feature
gate where it has one.
Related pages:
- Concepts defines the vocabulary used below (tenant, signal, segment, commit token, fold, snapshot).
- The consistency model holds the normative guarantees for acknowledgement and visibility.
- The server flag reference lists the server flags that change any behavior described here.
ravel-server serves HTTP on one listener (--listen-http, default
127.0.0.1:4318). OTLP ingest for metrics, logs, and traces is also served
over gRPC on a second listener (--listen-grpc). This page documents the
HTTP surface only.
A tenant-scoped route resolves the request to a tenant before it does any
work. The default resolver is a static bearer-token map (--tenant-token or
--tenant-token-file). A deployment can instead resolve the tenant from an
OIDC token or a proxy-forwarded mTLS identity. With every resolver, a request
to a tenant-scoped route that carries no resolvable credential is rejected
with 401 before any object-store access. A 401 therefore guarantees that
nothing was written or read.
The health and /metrics routes carry no tenant identity and require no
credential. They are served in every mode, including maintain mode, whose
router is otherwise empty.
Where a dedicated mutual-TLS listener is configured (--mtls-listener), it
serves the same ingest and query routes with the mTLS resolver. It serves
none of the four probe routes (/healthz, /readyz, /-/healthy,
/-/ready) nor /metrics. The one exception is
GET /api/v1/agents/health, which --agents mounts on both listeners; it
is unauthenticated there too and reports operator settings only. The other
/api/v1/agents routes are mounted on both listeners as well, each with
that listener's resolver (see Agents routes).
Served in all and gateway modes.
- Every ingest route is strict-acknowledgement by default: a 2xx means that the data object and its commit record are durably stored.
- On a strict acknowledgement, the OTLP responses carry an
x-ravel-commit-tokenheader: a comma-separated token per shard that the request flushed through. A later query replays the tokens asmin_commit_tokenfor read-your-write. - Buffered acknowledgement is opt-in per request, with the
x-ravel-ingest-mode: bufferedheader, on the OTLP routes only. - The same header picks a strict write's flush policy on the OTLP routes:
strict-arrivalopens the flush on arrival,strict-agedwaits for the age threshold or the size trigger.strict, an absent header, and any other value use the server's--strict-flush-policy(defaultaged). Both policies acknowledge on the same terms. - The request body limit is 16 MiB on the wire and 64 MiB after decompression, on every ingest route.
- OTLP HTTP accepts an absent, identity, or single
gzip(x-gzip)Content-Encoding. A chained or unknown encoding is 415. - Remote Write requires Snappy compression. It is strict-acknowledgement
only, ignores a buffered-mode header, and always uses the
--strict-flush-policyflush policy.
| Method | Path | Request | Response | Status codes | Credential | Modes | Feature |
|---|---|---|---|---|---|---|---|
| POST | /v1/metrics |
OTLP ExportMetricsServiceRequest protobuf, gzip optional |
OTLP ExportMetricsServiceResponse protobuf; x-ravel-commit-token on strict ack |
200, 400, 401, 413, 415, 429, 500, 503 | Yes | all, gateway |
none |
| POST | /v1/logs |
OTLP ExportLogsServiceRequest protobuf, gzip optional |
OTLP ExportLogsServiceResponse protobuf; x-ravel-commit-token on strict ack |
200, 400, 401, 413, 415, 429, 500, 503 | Yes | all, gateway |
none |
| POST | /v1/traces |
OTLP ExportTraceServiceRequest protobuf, gzip optional |
OTLP ExportTraceServiceResponse protobuf; x-ravel-commit-token on strict ack |
200, 400, 401, 413, 415, 429, 500, 503 | Yes | all, gateway |
none |
| POST | /api/v1/write |
Prometheus Remote Write 1.0 or 2.0 protobuf, Snappy-compressed | Empty body; the x-prometheus-remote-write-*-written count headers |
200, 400, 401, 413, 415, 429, 500, 503 | Yes | all, gateway |
none |
- A 429 carries a
Retry-Afterheader and sheds the request without buffering it. - Remote Write returns 415 when the content-type or version header names neither 1.0 nor 2.0.
- An active-series cap breach on Remote Write reduces the written count inside a 200. On Remote Write, 429 is reserved for rate limits.
Served in all and query modes. Every query route resolves one immutable
snapshot and answers from it. Commits, compactions, and deletions that land
mid-query cannot change the answer. A caller that holds commit tokens passes
them back as repeated min_commit_token values for read-your-write.
The Prometheus-shaped routes take form-encoded parameters: on GET, the query
string, and on POST, the body. /api/v1/sql, /api/v1/esql and
/api/v1/analytics take a JSON body.
| Method | Path | Request | Response | Status codes | Credential | Modes | Feature |
|---|---|---|---|---|---|---|---|
| GET, POST | /api/v1/query |
query, time, timeout, min_commit_token |
Prometheus instant-query JSON envelope | 200, 400, 401, 422, 500, 503, 504 | Yes | all, query |
none |
| GET, POST | /api/v1/query_range |
query, start, end, step, timeout, min_commit_token |
Prometheus range-query JSON envelope | 200, 400, 401, 422, 500, 503, 504 | Yes | all, query |
none |
| GET | /api/v1/labels |
start, end, match[] |
Prometheus label-names JSON envelope | 200, 400, 401, 422, 500, 503, 504 | Yes | all, query |
none |
| GET | /api/v1/label/{name}/values |
path name, plus start, end, match[] |
Prometheus label-values JSON envelope | 200, 400, 401, 422, 500, 503, 504 | Yes | all, query |
none |
| GET, POST | /api/v1/series |
match[], start, end |
Prometheus series JSON envelope | 200, 400, 401, 422, 500, 503, 504 | Yes | all, query |
none |
| GET | /api/v1/status/buildinfo |
none | Prometheus build-info JSON; Ravel's own version, empty goVersion |
200 | No | all, query |
none |
| GET | /api/v1/metadata |
metric, limit |
Prometheus metric-metadata JSON; an empty object when no tenant resolves | 200 | Optional | all, query |
none |
| GET, POST | /api/v1/query_exemplars |
query, start, end |
Prometheus exemplars JSON envelope | 200, 400, 401, 422, 500, 503, 504 | Yes | all, query |
none |
| POST | /api/v1/analytics |
JSON: query, start, end, step, op, optional timeout, min_commit_token, allow_partial |
JSON analytics envelope, one entry per series | 200, 400, 401, 422, 500, 503, 504 | Yes | all, query |
none |
| POST | /api/v1/sql |
JSON: query, start, end, optional timeout, min_commit_token (each <token> or <table>:<token>; qualified is required on a statement reading two or more signal tables) |
Arrow IPC stream or JSON, per Accept |
200, 400, 401, 422, 500, 503, 504 | Yes | all, query |
sql |
| POST | /api/v1/esql |
JSON: query, optional start, end, timeout |
JSON: typed columns, rows, stats, and warnings when there are any |
200, 400, 401, 422, 500, 503, 504 | Yes | all, query |
esql |
| GET | /api/traces/{trace_id} |
path trace_id, plus optional start, end, timeout, min_commit_token (<token> or spans:<token>) |
The /api/v1/sql response for every span of the trace: Arrow IPC stream or JSON, per Accept |
200, 400, 401, 422, 500, 503, 504 | Yes | all, query |
sql |
| POST | /mcp |
One MCP JSON-RPC message (revision 2026-07-28 or 2025-11-25) |
JSON or text/event-stream, per revision; a tool result carries the result envelope |
200, 400, 401, 403, 405, 413, 500 | Yes | all, query |
mcp |
| GET | /api/v1/agents/health |
none | JSON: feature, mode, activation_concurrency, poll_concurrency, memory_budget_bytes and the five quotas as --agents-* set them |
200 | No | all, query, and only with --agents; also on the mTLS listener |
agents |
| POST | /api/v1/agents/create |
JSON: agent (kind, key) |
JSON: agent_id, incarnation, created |
200, 401, 403, 404, 413, 422, 500, 503 | Yes, agents |
all, query, only with --agents; also on the mTLS listener |
agents |
| POST | /api/v1/agents/send |
JSON: agent, handler, request_id, payload |
JSON: operation, accept_seq, duplicate |
200, 401, 403, 404, 409, 413, 422, 429, 500, 503 | Yes, agents |
all, query, only with --agents; also on the mTLS listener |
agents |
| POST | /api/v1/agents/call |
JSON: the send body plus wait_ms |
JSON: operation, status, result (null until terminal) |
200, 401, 403, 404, 409, 413, 422, 429, 500, 503 | Yes, agents |
all, query, only with --agents; also on the mTLS listener |
agents |
| GET | /api/v1/agents/operation |
operation |
JSON: operation, status, accept_seq, blocked_effect |
200, 401, 403, 404, 422, 500, 503 | Yes, agents |
all, query, only with --agents; also on the mTLS listener |
agents |
| GET | /api/v1/agents/result |
operation, optional wait_ms |
JSON: operation, result (null until terminal) |
200, 401, 403, 404, 422, 500, 503 | Yes, agents |
all, query, only with --agents; also on the mTLS listener |
agents |
| POST | /api/v1/agents/resolve |
JSON: agent, handler, request_id |
JSON: operation and status, both null when no receipt matches |
200, 401, 403, 404, 413, 422, 503 | Yes, agents |
all, query, only with --agents; also on the mTLS listener |
agents |
| POST | /api/v1/agents/cancel |
JSON: operation |
JSON: operation, status, requested |
200, 401, 403, 404, 413, 422, 500, 503 | Yes, agents |
all, query, only with --agents; also on the mTLS listener |
agents |
| GET | /api/v1/agents/state |
kind, key |
JSON: agent_id, incarnation, state_version, content_type, state, active_turn (absent when no turn is open), mailbox (each accept_seq, handler, source, accepted_unix_ns, failed_attempts; absent when empty), usage (receipts and max_receipts, the sealed pages results_pages, events_pages and timers_pages and max_index_pages, head_bytes and max_head_bytes; agents_timers_e2e::a_sweep_cycle_renders_the_fill_families_and_the_refusal_counter) |
200, 401, 403, 404, 422, 503 | Yes, agents |
all, query, only with --agents; also on the mTLS listener |
agents |
| POST | /api/v1/agents/events |
JSON: agent, cursor (null for the start), limit, optional operation |
JSON: events, next_cursor, terminal (only on the page that drains a finished operation) |
200, 401, 403, 404, 410, 413, 422, 503 | Yes, agents |
all, query, only with --agents; also on the mTLS listener |
agents |
| POST | /api/v1/agents/timers |
JSON: agent, timer_key, due_unix_ns (decimal string), handler, payload |
JSON: timer, generation |
200, 401, 403, 404, 409, 413, 422, 500, 503 | Yes, agents |
all, query, only with --agents; also on the mTLS listener |
agents |
| POST | /api/v1/agents/timers/cancel |
JSON: agent, timer_key |
JSON: outcome (cancelled, already_fired, unknown) |
200, 401, 403, 404, 409, 413, 422, 500, 503 | Yes, agents |
all, query, only with --agents; also on the mTLS listener |
agents |
| GET | /api/v1/agents/timers |
kind, key |
JSON: timers (each timer, timer_key, generation, due_unix_ns, handler, status) |
200, 401, 403, 404, 422, 503 | Yes, agents |
all, query, only with --agents; also on the mTLS listener |
agents |
| GET | /api/v1/agents/inspector |
none | text/html; charset=utf-8: the embedded inspector page |
200, 401, 403, 503 | Yes, agents |
all, query, only with --agents; also on the mTLS listener |
agents |
| POST | /api/v1/agents/worker/register |
JSON: handlers (each handler, version, digest) |
JSON: worker_id |
200, 401, 403, 409, 413, 422, 503 | Yes, agents_worker |
all, query, only with --agents; also on the mTLS listener |
agents |
| POST | /api/v1/agents/worker/poll |
JSON: worker_id, handlers, wait_ms |
JSON: assignment (lease, agent, operation, accept_seq, handler, state, payload, steps), null when nothing is due |
200, 401, 403, 404, 409, 413, 422, 500, 503 | Yes, agents_worker |
all, query, only with --agents; also on the mTLS listener |
agents |
| POST | /api/v1/agents/worker/commit |
JSON: lease as polled, outcome (completed with state, output, events, timers, intents; or failed with error, terminal) |
JSON: operation, status, state_version, rebased, effects |
200, 401, 403, 404, 409, 413, 422, 500, 503 | Yes, agents_worker |
all, query, only with --agents; also on the mTLS listener |
agents |
| POST | /api/v1/agents/worker/heartbeat |
JSON: lease as polled |
JSON: lease_expires_unix_ns |
200, 401, 403, 404, 409, 413, 422, 503 | Yes, agents_worker |
all, query, only with --agents; also on the mTLS listener |
agents |
| POST | /api/v1/agents/worker/step |
JSON: lease as polled, step_key, tool_name, tool_version, policy (provider_idempotency, at_least_once, manual), input_hash (64 lowercase hex) |
JSON: step (a steps entry), effect, attempt, failure (null unless the recorded result is a tool failure) |
200, 401, 403, 404, 409, 413, 422, 500, 503 | Yes, agents_worker |
all, query, only with --agents; also on the mTLS listener |
agents |
| POST | /api/v1/agents/worker/effects/complete |
JSON: effect, attempt, result |
JSON: effect, status, late |
200, 401, 403, 404, 409, 413, 422, 500, 503 | Yes, agents_worker |
all, query, only with --agents; also on the mTLS listener |
agents |
| POST | /api/v1/agents/worker/effects/fail |
JSON: effect, attempt, error |
JSON: effect, status, late |
200, 401, 403, 404, 409, 413, 422, 500, 503 | Yes, agents_worker |
all, query, only with --agents; also on the mTLS listener |
agents |
| GET | /api/v1/agents/effects |
kind, key |
JSON: effects (each effect, turn_id, tool_name, tool_version, policy, status, attempt) |
200, 401, 403, 404, 422, 503 | Yes, agents |
all, query, only with --agents; also on the mTLS listener |
agents |
| GET | /api/v1/agents/effects/{effect} |
none | JSON: the effects entry plus step_key, result_attempt, result, failure (present only when the result is a tool failure), attempts (each attempt, status), evidence (each attempt, result, and failure when the report was one) |
200, 401, 403, 404, 422, 503 | Yes, agents |
all, query, only with --agents; also on the mTLS listener |
agents |
| POST | /api/v1/agents/reconcile |
JSON: effect, attempt, decision (record_result with result, or retry) |
JSON: effect, status, attempt |
200, 401, 403, 404, 409, 413, 422, 500, 503 | Yes, agents_admin |
all, query, only with --agents; also on the mTLS listener |
agents |
| POST | /api/v1/agents/sweep |
none | JSON: wake_list_pages, dir_list_pages, head_gets, hints_read, hints_deleted, timers_fired, agents_loaded, dir_entries_repaired |
200, 401, 403, 500, 503 | Yes, agents_admin |
all, query, only with --agents; also on the mTLS listener |
agents |
| POST | /api/v1/agents/destroy |
JSON: agent (kind, key), optional incarnation |
JSON: agent_id, incarnation, destroyed, abandoned (inputs, active_turn, effects_in_doubt, effects_authorized, timers_scheduled) |
200, 401, 403, 404, 413, 422, 500, 503 | Yes, agents_admin |
all, query, only with --agents; also on the mTLS listener |
agents |
Both routes exist for Prometheus-shaped clients. Grafana's datasource test
probes both. /api/v1/metadata never returns 401. A request with no
resolvable tenant receives an empty object, and reads no object storage on
that path.
The op type field selects one of two operations: change point detection
(change_point) and summary statistics (summary).
| Condition | Status |
|---|---|
An unknown op, a missing field, or a malformed body |
400 |
| An analytics computation error, or the per-call series cap | 422 |
Partial federated coverage without allow_partial: true |
503 |
/api/v1/analytics and /api/v1/query_exemplars take a permit from the same
query concurrency ceiling as the other query routes. A request that the
ceiling refuses gets the same 503 body that those routes return.
/api/v1/sql is behind the sql cargo feature. The published server image
builds that feature, so the route is available there.
The SQL surface registers five tables, one per signal: samples (metrics),
logs, spans (traces), alerts (alert state transitions), and audit
(audit records, including the query-audit trail).
A statement may name several of those five and the tenant's Parquet tables,
for example a join of samples and logs. Each signal table is read from
its own snapshot, with no order between them, and the join runs on the
receiving server under a memory ceiling that is best-effort for join
operators. In a statement reading two or more tables, a relation alias,
derived-table alias or CTE name equal to one of its other tables' names or
to any of the five signal table names is 400 before any catalog listing. An
alias equal to its own table's name (FROM logs AS logs) is admitted, and a
one-table statement is not checked. See
the query guide for the terms.
Request limits:
- The request body is capped at 64 KiB. A larger body is 400.
- A statement that carries more than 1,000 tokens is 400. The count covers tokens outside string literals and comments. The message names the measured count and the maximum.
- The token gate runs before the statement is parsed. It bounds the depth of the expression tree that a statement can build, so it constrains structure and not text length.
- A string literal costs one token, whatever the length of its payload. An identifier or a number costs one token, whatever its length. A comment costs nothing.
- The gate applies to Flight SQL too, where it surfaces as
InvalidArgument.
See docs/query-engine.md, "SQL statement complexity gate".
min_commit_token is a list. Each entry is a commit token, or a token
qualified with the table it bounds, <table>:<token> (split once at the first
colon). Each qualified token bounds only that table's snapshot. Each of these
is 400 bad_data before any catalog or object-store request:
- An unqualified token on a statement that reads two or more of
samples,logs,spans,alertsandaudit. The message names those tables in that order:qualify min_commit_token with one of samples, logs, .... - A qualifier that is not, exactly and in lower case, the name of a signal table the statement reads. The message quotes the qualifier, cut to 64 bytes.
- Any token on a statement that reads only Parquet tables.
- An empty qualifier, or a part after the colon that is not a token
(
invalid min_commit_token).
See qualifying a token by table.
The JSON response envelope has one array per row under data.rows:
{"status": "success",
"data": {"columns": [{"name": "ts", "type": "..."}], "rows": [[...]]},
"stats": {}}stats holds accounting, estimate, phases, signals, io,
timings and pruning, and cache when the fetcher cache is on (see
per-statement cache counters).
signals has one entry per signal table the
statement reads, in the order samples, logs, spans, alerts,
audit, each with exactly table, segments, segmentsPruned and
phases (the table's own cost, in the shape of stats.phases). Summed
over the entries, each figure of phases equals the statement's
stats.phases figure, less what Parquet tables cost; a Parquet table has no
entry. The
query engine reference defines
every section.
JSON encoding per column type:
- A non-finite float is a string:
NaN,+Inf, and-Inf. - Every integer width is a JSON number.
Float16is widened to a 64-bit float and follows the float rule above, including theNaN,+Infand-Infstrings.- A timestamp of any unit is an integer count of nanoseconds since the
epoch, with its time zone dropped. A value past 2^53 needs a client that
parses JSON integers exactly, which a JavaScript
JSON.parsedoes not. - A
Time32orTime64of any unit is an integer count of nanoseconds since midnight. - A
Durationof any unit is a signed integer count of nanoseconds. Date32andDate64areYYYY-MM-DDstrings.- Every decimal width (
Decimal32,Decimal64,Decimal128,Decimal256) is a string that holds its exact decimal text. - Binary columns are lowercase hex strings.
- An
Intervalof any unit is an object with all three integer fieldsmonths,daysandnanoseconds, each 0 where the unit has no such component. The milliseconds of a day-time interval become nanoseconds. The three fields stay apart because none converts exactly into another: a month has no fixed number of days. Also, an ISO 8601 duration cannot carry a negative component beside a positive one. - Every list layout (
List,LargeList,FixedSizeList,ListView,LargeListView) is a JSON array whose elements follow these same rules. - A
Dictionarywith any integer key type (Int8toInt64,UInt8toUInt64) is encoded as the value that its key addresses.
A column whose type or value has no JSON encoding fails the query with 422
execution. The cases are:
- a type with no rule above, which is also logged at
warn - a timestamp or a duration past the i64 nanosecond range
- a time that is negative, or a whole day or more
- a date outside years 0000 to 9999
- a
Date64that is not a whole day
The message names the column and its Arrow type and gives the reason, never
the value:
column "<name>" of type <type> cannot be encoded as JSON: <reason>; request the Arrow IPC format to read it exactly.
To get an Arrow IPC stream instead, send
Accept: application/vnd.apache.arrow.stream. The stream is bit-exact for
every type and reads every such column.
/api/v1/esql runs one ES|QL statement over the logs table. It is behind
the esql cargo feature, which turns on sql too. The published server
image does not build esql, so the route is absent there, and a build
without the feature answers 404. A build with it serves the route on the
main HTTP listener and on the mTLS listener, each under its own credential.
The statement runs through the same admission ceiling, deadline, usage
record, query audit record and cost recording as /api/v1/sql. See
ES|QL queries for the language that runs.
Request body, JSON:
| Field | Required | Meaning |
|---|---|---|
query |
Yes | The statement text. |
start |
No | Unix seconds, a float. The window starts no earlier than this, inclusive. |
end |
No | Unix seconds, a float. The window ends before this, exclusive. |
timeout |
No | Seconds. It can only lower the server deadline. |
start and end narrow the window the statement's own @timestamp
bounds give it; they never widen it. Any other field is 400. params is
refused by name: the route runs no parameterised statement, so write each
value into the statement. The route takes no min_commit_token. The body is
capped at 64 KiB.
curl -s -X POST http://127.0.0.1:4318/api/v1/esql \
-H "Authorization: Bearer demo-token" \
-H "Content-Type: application/json" \
-d '{"query": "FROM logs | WHERE @timestamp >= NOW() - 1 hour | STATS errors = COUNT(*) WHERE log.level == \"ERROR\" BY service.name | SORT service.name"}'The response is always JSON. Here, with the row values illustrative and the
stats sections shown empty for brevity:
{"status": "success",
"data": {"columns": [{"name": "errors", "type": "long"},
{"name": "service.name", "type": "keyword"}],
"rows": [[3, "api"], [2, "web"]]},
"stats": {"accounting": {}, "estimate": {}, "io": {}, "phases": [],
"pruning": {}, "timings": {}}}- Each column's
typeis its ES|QL type:keyword,long,double,boolean,date_nanosornull. - A
date_nanosvalue is an integer count of nanoseconds since the Unix epoch. statshas the sections/api/v1/sqlreports for a statement that reads the catalog, exceptsignalsandcache:accounting,estimate,io,phases,pruningandtimings. AROWstatement reads no object, and itsstatsholds onlyaccountingandestimate, with no request counted.warnings, an array of strings, is present only when there is something to say. Today that is one case: the statement's ownLIMITwas above 10,000 and was lowered to it, as inthe statement's LIMIT 20000 was lowered to 10000, the most rows an ES|QL query returns.
Errors use the shared body, {"status": "error", "errorType": ..., "error": ...}. The status follows the shared mapping in
Query status codes, with these ES|QL cases:
| Condition | Status | errorType |
|---|---|---|
The statement does not parse, or passes a structural limit; the message starts line L:C: and states no byte offset |
400 | bad_data |
The statement parses and does not bind: an unknown column, a type mismatch, an aggregate outside STATS |
400 | bad_data |
A command, source, function, operator or type this build does not run; the message starts line L:C: and names the construct, as shown below |
400 | bad_data |
params, an unknown field, a body that is not JSON, or start after end |
400 | bad_data |
| A bound statement the planner cannot plan; the message does not carry the planner's text | 422 | execution |
| A statement whose time window is wider than the catalog serves | 422 | execution |
| A failure while the statement runs | As for /api/v1/sql |
As for /api/v1/sql |
The refusal of a construct this build does not run reads, for FROM metrics:
line 1:6: source [metrics] is not supported
A statement that does not parse is refused before it takes a permit from the admission ceiling and before any object-store read, and it writes no audit or usage record.
/api/traces/{trace_id} is behind the sql cargo feature. It runs
SELECT * FROM spans WHERE trace_id = X'<trace_id>' ORDER BY start_ts, span_id through the /api/v1/sql path, with the same envelope,
authentication, limits, deadline and query audit record.
trace_idis 32 hexadecimal characters in either case, after the path segment is percent-decoded once. Anything else is 400bad_data, before any object-store read.- A trace the tenant does not hold is 200 with zero rows.
- Without
start, the lookup reads every span the tenant retains, not the one hour beforeendthat/api/v1/sqldefaults to. An omittedendis the current time. startandendchoose which objects are read and do not filter rows, so a span that starts outside them can still be returned.min_commit_tokenmay repeat. Each value is a token, or a token qualifiedspans:<token>, the one table the lookup reads. Any other qualifier is 400bad_databefore any object-store read.
/mcp is behind the mcp cargo feature and --mcp. A build that carries
the feature serves no MCP route until an operator passes the flag. /mcp is
the only MCP path, and it serves POST only.
Every request is authenticated with the listener's own tenant resolver before the JSON-RPC message is parsed.
| Condition | Status |
|---|---|
No resolvable credential, whatever the request asked for, including a legacy initialize |
401 |
An Origin outside --mcp-allowed-origins |
403 |
A body past --mcp-max-body-bytes |
413 |
On revision 2026-07-28, a Mcp-Method or Mcp-Name header that disagrees with the body |
400 |
The /api/v1/agents routes are behind the
agents cargo feature and --agents. Both listeners mount them, each with
its own tenant resolver, and the tenant always comes from the credential,
never from the request (agents_e2e::capabilities_are_enforced_on_both_listeners).
Client routes need the principal's agents capability, worker routes its
agents_worker capability, and reconcile, sweep and destroy its
agents_admin capability. Every handler admits its capability before
anything else, and a credential without it gets 403 with a body naming the
capability: the server's route table carries each route's capability, and
agents::tests::every_route_refuses_a_token_lacking_its_capability sends
every route in it a token holding every capability but that one
(end to end: agents_e2e::capabilities_are_enforced_on_both_listeners for
agents and agents_worker,
agents_effects_e2e::manual_blocks_until_reconciled_and_audits_the_decision_once,
agents_timers_e2e::sweep_route_counts_match_the_store_and_needs_agents_admin
and
agents_effects_e2e::destroy_fences_the_lease_holder_audits_once_and_a_recreate_starts_incarnation_two
for agents_admin). The router answers 404 for any path outside the table
above (agents_e2e::a_path_outside_the_route_table_is_404).
The router carries its own admission counter, one
lane for the waiting routes (call, result, worker/poll) sized by
--agents-poll-concurrency and one for every other route sized by
--agents-activation-concurrency. It never takes a query permit, so a full
agents lane leaves query routes serving
(agents_e2e::admission_refusal_is_503_and_takes_no_query_permit).
Every refusal after authentication is an ErrorBody: code, retry,
reason, and ids holding only safe identifiers (agent id, incarnation,
operation, turn, step, effect, timer). Payloads and state never appear in
an error. A budget_exceeded or mailbox_full refusal of an agent's
lifetime cap adds cap, naming it: receipts, index_pages,
head_bytes, mailbox or timers
(agents_timers_e2e::a_sweep_cycle_renders_the_fill_families_and_the_refusal_counter
answers a send past the receipt cap 413 with cap receipts).
code |
Status | retry |
|---|---|---|
agent_gone, including an unknown operation handle, whose ids.operation echoes the handle (agents_e2e::unknown_operation_handle_is_agent_gone_with_the_handle) |
404 | never |
idempotency_conflict, ownership_lost, incompatible_handler, non_deterministic_replay |
409 | never |
effect_in_doubt |
409 | reconcile |
outcome_unknown; resolve by request_id before resending (agents_e2e::resolve_after_a_lost_send_ack_returns_the_same_operation) |
500 | resolve_operation |
outcome_unknown from reconcile, which mints no operation handle. For record_result, send the same decision again: a repeat of a decision that took effect answers the same body. For retry, do not resend: read GET /api/v1/agents/effects/{effect} and follow "Reconciling an effect in doubt" in the operations guide, since no process holds the attempt and a repeated retry is refused (agents_effects_e2e::a_reconcile_whose_head_write_times_out_answers_the_reconcile_hint) |
500 | reconcile |
outcome_unknown from destroy, which mints no operation handle, with ids.incarnation the incarnation it read: send the same destroy again naming that incarnation (agents_effects_e2e::a_destroy_whose_head_write_is_lost_answers_backoff_and_a_retry_answers_destroyed_false) |
500 | backoff |
mailbox_full |
429 | backoff |
budget_exceeded, including a body past the router's limit (agents_e2e::send_over_the_body_limit_is_413_with_a_typed_body) |
413 | never |
store_unavailable, including the admission refusal agents admission limit reached |
503 | backoff |
unsupported_format, including a body or query string that does not parse |
422 | never |
No credential is 401 and a missing capability is 403, both in the query routes' plain error shape.
Long polls. result and worker/poll hold the request until there is
something to return or wait_ms elapses, and call waits the same way
after accepting. wait_ms is capped at 20 s on every one of them; a larger
value waits 20 s. A long poll that ends empty answers 200 with a null
result or assignment. A waiter answers as soon as the commit lands, not
at the next poll interval
(agents_e2e::send_then_result_through_real_server_survives_worker_replacement).
Event cursor. events returns committed events strictly after
cursor, in event sequence order, at most limit per page. A limit
above 1000 reads 1000, and a limit of 0 is 422 unsupported_format
(events limit must be at least 1; same test as below). The cursor is the decimal committed event sequence; null starts
at the beginning. next_cursor is the last sequence returned, or the input
cursor when the page is empty, so resuming from it never repeats or skips
an event (agents_e2e::events_page_after_the_durable_cursor_without_duplicates).
A cursor that is not a decimal sequence is 422. With operation set (an
operation handle of agent), a page holds only the events that
operation's turns committed, and the page that holds its last event once
it is finished carries terminal, its final status; no later page holds
an event of it. A cursor below the oldest retained event is 410 with
code cursor_expired, the cursor sent, oldest_cursor and restart
restart_from_oldest_cursor: read again from oldest_cursor
(agents_timers_e2e::events_resume_from_a_cursor_to_terminal_and_expire_below_the_window,
which also pins the resumed pages' exact sequences and the 422 for a
limit of 0). This release never trims events, so the 410 is the
contract for a later one that retains a window.
Timers. timers arms the agent's timer timer_key at due_unix_ns,
in nanoseconds since the Unix epoch as a decimal string, and answers the
timer id and its generation. Scheduling the same key again bumps the
generation and supersedes the earlier one, so only the new generation
fires, once
(agents_timers_e2e::a_reschedule_supersedes_the_earlier_generation).
timers/cancel answers cancelled before the firing, already_fired once
the firing committed (its input is then in the mailbox, which GET state
lists with source.kind timer), and unknown for a key never scheduled
(agents_timers_e2e::cancel_after_fire_is_already_fired_and_before_fire_is_cancelled).
GET timers lists the latest generation of each key as scheduled,
delivered or cancelled. A due timer fires from the sweep alone; the
_wake hint written beside it only speeds that up, and a timer whose hint
write failed is still delivered once, by a later process over the same
store
(agents_timers_e2e::due_timer_is_delivered_once_across_a_coordinator_restart_with_no_hint).
Sweep. Each tenant's coordinator runs a sweep pass every
agents.sweep_interval_ms (5000 by default; 0 runs no periodic pass, and
worker polls and sweep still run one), for the tenants this process has
served since it started. sweep runs one pass over the caller's tenant on
demand under the same budgets and answers what it did: the _wake/ and
_dir/ LIST pages, HEAD reads, hints read and deleted, timers fired, agents
loaded and _dir entries repaired. The counts are the requests the store
served for that pass, and the ravel_agents_sweep_* families on
/metrics move by the same amounts
(agents_timers_e2e::sweep_route_counts_match_the_store_and_needs_agents_admin).
The periodic pass needs no poll and no sweep call
(agents_timers_e2e::the_interval_sweep_runs_a_pass_per_interval_with_no_poll).
An idle worker poll runs a pass at most once per sweep_interval_ms, or
once per 5000 ms when it is 0, so idle polls never sweep on every wake
(agents_timers_e2e::interval_zero_runs_no_periodic_pass_and_polls_sweep_at_the_floor).
GET /api/v1/agents/health reports sweep_interval_ms.
Inspector. GET inspector serves one embedded HTML page with no
external asset and a content security policy that admits no origin but
this server. The route needs agents like any other, so a plain browser
navigation, which sends no bearer token, is 401; the page itself asks for
the token and reads state, timers, effects, effects/{effect} and
events with it. It shows the state version, the active turn, the
mailbox, the timers, the effect-backed steps and the effects with their
status, and for a blocked agent its reason and the action that unblocks
it. Hashes and raw payloads sit under <details>, and every payload
preview is labelled "preview"
(agents_timers_e2e::inspector_is_served_to_agents_tokens_with_every_section,
agents::tests::inspector_serves_the_embedded_page_as_html).
Workers. worker/register mints a fresh worker id per call, so a
worker process registers once at start and a restarted process gets a new
id; no two processes can share one. A worker polls with its id and handler
pins, and worker/commit and worker/heartbeat echo the polled lease.
The commit is fenced by that lease alone: once another worker has taken the
agent over, the old lease's commit is 409 ownership_lost and changes
nothing (agents_e2e::send_then_result_through_real_server_survives_worker_replacement).
The polled lease also carries recorded_unix_ns and random_seed, fixed
when the turn begins, so a replacement worker re-runs the turn with the
same clock and randomness. call, cancel and worker/heartbeat are
covered by agents_e2e::call_heartbeat_and_cancel_through_real_server.
Several server processes may serve one tenant: every write is conditional
on HEAD, so the guarantees hold, but a send accepted by a process with no
waiting worker is found by another process's sweep after a delay, and a
reconcile retry is handed over only within the process that answered it
(see docs/guides/operations/agents.md).
Steps and effects. A worker journals each tool call with
worker/step before running it, under the turn's lease. Steps are
sequential: a second step while one's attempt is still open is 409
idempotency_conflict. A step with a recorded result answers it without a
write: step.status completed with its output, or failure set when the
result is a tool failure reported through worker/effects/fail. A failure
leaves the step started with a null output, in worker/step and in a
polled assignment's steps alike, so a retried turn replays the failure
rather than an output, and worker/commit may then fail the turn
(agents_effects_e2e::a_failed_effect_replays_as_the_steps_failure_and_the_turn_may_fail).
A failure is typed by how worker/effects/fail records it, never by the
output's shape: an output of exactly {"error": "x"} reported through
worker/effects/complete replays as that output
(agents_effects_e2e::an_output_shaped_like_a_failure_replays_as_the_output).
Otherwise attempt is the attempt the caller may run and effect the
handle to report it under. A replay whose input_hash differs from the
journaled one is 409 non_deterministic_replay, and the operation reads
blocked
(agents_effects_e2e::a_drifted_replay_is_non_deterministic_and_blocks_the_turn).
A step whose input is being cancelled is 409 ownership_lost and no new
attempt is authorized
(agents_effects_e2e::cancel_during_an_open_attempt_refuses_the_replay_and_the_result_is_late).
When a worker dies holding an open attempt, the replacement's replay of the
step follows the step's policy. provider_idempotency hands out the same
attempt again, so whichever result arrives first is the result
(agents_effects_e2e::provider_idempotency_redispatches_the_same_attempt_and_the_late_result_counts).
at_least_once authorizes the next attempt; a result for the earlier one
then answers late: true, is kept as evidence and never becomes the result
(agents_effects_e2e::completion_from_a_dead_worker_attempt_records_the_effect_and_the_new_owner_commits_the_turn).
manual latches the effect in doubt: the step is 409 effect_in_doubt,
the operation reads blocked with blocked_effect set, and the turn may
neither continue nor fail until an agents_admin caller reconciles it. A
repeat of the same result for an attempt is accepted, and a different one
is 409 idempotency_conflict
(agents_effects_e2e::a_conflicting_final_result_for_the_same_attempt_is_refused).
effects lists an agent's journaled effects and effects/{effect}
inspects one: each attempt as open, in_doubt, result or
superseded, the result and the attempt it came from, and the late results
kept as evidence
(agents_effects_e2e::completion_from_a_dead_worker_attempt_records_the_effect_and_the_new_owner_commits_the_turn).
attempts lists at most the latest 64 attempts
(agents_timers_e2e::effect_inspect_lists_at_most_the_latest_64_attempts).
Evidence is held by the serving process and is not durable; a restart
forgets it, and so does a reconcile that settles the effect out of doubt.
A process names the evidence of at most 4096 effects per tenant; when that
is full, an effect whose result was recorded or whose input finished gives
its slot to a new one, and an effect still in doubt keeps it
(agents_effects_e2e::settled_effects_give_their_evidence_slot_to_an_effect_in_doubt).
Reconcile. reconcile settles an effect in doubt: record_result
attaches an operator-verified result, and retry authorizes one more
attempt. attempt must name the effect's latest attempt, else 409
idempotency_conflict; repeating a decision that already took effect
answers the same body and changes nothing. retry dispatches nothing: it
answers the authorized attempt's number in attempt, and the turn's owner
learns it from its next worker/step for the step, which answers that
attempt for the worker to run instead of refusing it as open. Until its
result is reported, a further step is 409 idempotency_conflict and a
retry naming that attempt is 409 ownership_lost, since it is not in
doubt
(agents_effects_e2e::a_reconcile_retry_hands_the_new_attempt_to_the_owners_next_step).
The next worker/step for the step gets the attempt whichever worker
holds the turn by then, including one that took the turn over after the
owner's lease expired
(agents_effects_e2e::a_reconcile_retry_is_handed_to_the_owner_after_a_takeover).
The hand-over is kept only in the memory of the server process that
answered the reconcile: if that process restarts before the step, or
another server process serves the step, the attempt reads as lost and a
manual effect is back in doubt, so reconcile it again. For the same
reason a repeated retry is answered as applied only while that process
holds the hand-over unclaimed. A reconcile that answers 500
outcome_unknown hands its attempt to no process. The effect's status
may still read authorized in GET /api/v1/agents/effects/{effect}
whether or not another reconcile's process holds the attempt, and the
attempt is dispatched at most once either way
(concurrent_reconciles_on_two_coordinators_dispatch_the_attempt_at_most_once).
If the effect is in doubt again after the owner's next step, the attempt
may have run, so resolve it as any effect in doubt; the operations guide's
"Reconciling an effect in doubt" has the procedure
(a_reconcile_authorization_with_a_lost_ack_is_not_handed_over_and_latches_in_doubt_again).
A retry naming the attempt before one a
worker's own step authorized, such as an at_least_once attempt raised
after a takeover, is 409 ownership_lost and writes no audit record
(agents_effects_e2e::a_reconcile_retry_of_an_attempt_a_takeover_raised_is_ownership_lost).
Before changing an effect in
doubt the server submits one audit record with kind agents_reconcile
to the audit pipeline, which writes it into the tenant's audit shard,
naming the effect, attempt, decision and the caller's tenant and
capabilities. The record names agents.principal.subject and
agents.principal.subject_source when the caller's token or claim carries a
subject (a subject=LABEL entry on its --tenant-token pair, or the claim
--oidc-subject-claim names); two admins whose tokens carry none are not
told apart. No response names the subject, and no request header other than
the credential sets it
(agents_effects_e2e::reconcile_and_destroy_records_name_each_admins_configured_subject_and_a_token_without_one_writes_no_subject_attr).
Without --audit-mode no record is written. Only
--audit-mode required refuses on a failed audit write: the reconcile is
then 503 store_unavailable and nothing changes
(agents_effects_e2e::manual_blocks_until_reconciled_and_audits_the_decision_once).
Under --audit-mode best-effort the pipeline reports a failed write as
written, so the reconcile proceeds without its record. A reconcile that
repeats a decision already taken writes no second record
(agents_effects_e2e::a_reconcile_retry_hands_the_new_attempt_to_the_owners_next_step).
Destroy. destroy tombstones the agent's current incarnation in one
HEAD write and answers what the tombstone abandoned: inputs is the
accepted inputs never consumed, the active turn's input among them while a
turn runs; active_turn whether a turn was open; effects_in_doubt and
effects_authorized the effects in each state; timers_scheduled the
timers still scheduled. After it, every route that names the agent or a
handle minted under that incarnation answers 404 agent_gone: state,
operation, result, reconcile, and the lease holder's
worker/heartbeat, effect result and worker/commit. A result parked on
one of the agent's operations ends inside its request with agent_gone,
woken by the destroy rather than by its deadline
(agents_effects_e2e::destroy_fences_the_lease_holder_audits_once_and_a_recreate_starts_incarnation_two).
Destroying an agent that is already tombstoned answers 200 with
destroyed false and the counts of the tombstoned HEAD, writes nothing and
writes no audit record (the same test). An agent that never existed, and an
incarnation that is not the current one, are 404 agent_gone with
nothing written and nothing audited; the refusal names the pinned
incarnation in ids.incarnation
(agents_effects_e2e::destroy_of_an_unknown_agent_or_a_stale_incarnation_is_agent_gone_and_not_audited).
create on a destroyed agent starts incarnation 2 with an empty mailbox,
receipt list and state; a handle from incarnation 1 stays agent_gone, and
a request_id used in incarnation 1 is accepted as new (the first test).
Before the tombstone write the server submits one audit record with kind
agents_destroy to the audit pipeline, naming the agent id, the
incarnation and the abandoned counts as it read them
(audit::tests::the_destroy_event_names_tenant_agent_incarnation_counts_and_caller).
The agents_destroy record names the caller's tenant and capabilities. The
record names agents.principal.subject and agents.principal.subject_source
when the caller's token or claim carries a subject; two admins whose tokens
carry none are not told apart, as for reconcile. The response's counts come from the HEAD the
tombstone was applied to, so a write landing between the read and the
tombstone shows in the response and not in the record. Without
--audit-mode no record is written. Under --audit-mode required a failed
audit write refuses the destroy with 503 store_unavailable and the agent
stays active in its incarnation
(agents_effects_e2e::a_destroy_whose_audit_record_is_refused_leaves_the_agent_active);
under --audit-mode best-effort the pipeline reports a failed write as
written, so the destroy proceeds without its record. A destroy whose
tombstone write has an unknown outcome is 500 outcome_unknown with retry
backoff and the incarnation it read in ids.incarnation. Send the same
destroy again with that incarnation: if the first write landed, the
retry answers 200 with destroyed false, and the one audit record is the
first attempt's
(agents_effects_e2e::a_destroy_whose_head_write_is_lost_answers_backoff_and_a_retry_answers_destroyed_false).
Pinning the incarnation keeps the retry from destroying an incarnation a
create started in between.
For the query routes, the status codes come from one shared error mapping.
- 400
bad_data: a malformed query, a bad time range, or a non-positive step. On/api/v1/sqlit also covers amin_commit_tokenthe statement cannot take: unqualified on a statement that reads two or more signal tables, qualified with a table the statement does not read, or on a statement that reads only Parquet tables (see/api/v1/sql). - 401
unauthorized: no resolvable credential. - 422
execution: one of these.- A resource-budget refusal: too many segments, series, samples, or scanned bytes, or an over-wide window.
- An unsupported construct.
- An SQL result column that the JSON encoding cannot represent. See
/api/v1/sql. Arrow IPC reads it exactly.
- 500
internal: a permanent data-integrity fault in already-stored objects. It is not retryable. Its message is fixed, so no object key or tenant hash leaks. The faults are:- A corrupt segment.
- An unreconstructable or mismatched commit record.
- A catalog object (commit, compaction or rewrite record, erasure request,
HEAD, snapshot part or postings) that does one of these:
- its stored bytes fail to decode at a format version that this build covers
- it carries a format version below the lowest that this build supports
- it leaves an enum field unset (proto3's default 0) where a value is required
- An erasure request or rewrite record that names a signal this build does not know.
- A provisioning record that fails to decode, is misfiled, carries a corrupt shard-generation or format-floor history, or carries a format version below the lowest that this build supports.
- A supersession chain of compaction or rewrite records that is cyclic, is deeper than the resolver's fixed bound, or names a predecessor with a different input set.
- A non-monotonic run.
- 503
unavailable: retryable. The causes are:- A transient storage fault.
- An invalidated snapshot.
- A catalog object (commit, compaction or rewrite record, erasure request, HEAD, snapshot part or postings) written in a format version above the highest that this build reads, or one that carries an enum value above the highest that it reads (a snapshot part entry level). A peer on a newer build can read such an object during a rolling upgrade.
- A provisioning record written in a format version above the highest that this build reads.
- A Parquet table's manifest or grants record written in a format version above the highest that this build reads.
- An unsatisfiable
min_commit_token, qualified or not. The message is fixed and names no table.
- 504
timeout: the query passed its deadline.
These faults answer 500, not 503:
- A store fault that is a checksum mismatch. This applies on a catalog read, on a segment fetch, and on a SQL read of a Parquet table's manifest, grants record or data file.
- A Parquet table's manifest or grants record written in a format version below the lowest that this build supports.
These faults answer no error:
- A column-statistics object that this build cannot decode. The query reads the data instead.
- A decode job for a catalog snapshot part, postings or column-statistics object that panics, or that the read CPU gate cancels at shutdown. The query still answers exactly, by a slower path.
Served in all and query modes, alongside the query surface, because it
shares the same catalog and folder identity as the scheduled fold.
| Method | Path | Request | Response | Status codes | Credential | Modes | Feature |
|---|---|---|---|---|---|---|---|
| POST | /api/v1/admin/fold |
JSON: signal, optional tenant |
JSON naming the fold outcome | 200, 400, 401, 403, 503 | Yes | all, query |
none |
/api/v1/admin/fold triggers a catalog fold for one tenant and one signal.
It is the on-demand form of the background fold. The same tenant credential
that the query routes require authorizes it. When the body names a tenant,
that name must hash to the authenticated tenant, or the request is 403. A
fold reveals no data and destroys none: it rewrites a query-cost index that
the tenant already owns.
The call returns 200 with one of four named outcomes:
published: this call wrote a new snapshot and advanced HEAD.nothing_eligible: no commit was eligible. An ingest hour seals only after the maximum flush lifetime plus a clock-skew allowance plus a fold safety margin has elapsed. A fold run right after a load therefore seals nothing.lost_cas: a concurrent fold won the HEAD compare-and-swap. The catalog is fine and the winner's snapshot is published. This call did not publish one.throttled: the rate gate declined because HEAD was published more recently than the fold interval. No listing ran, so this call makes no eligibility claim.
A 503 on this route means that the outcome is unknown. Retry the call. A 503 never means that nothing was written.
Served in all and query modes: the modes that build a query engine, and
therefore the modes that an alert evaluator runs in. A gateway or
maintain process serves 404 here.
| Method | Path | Request | Response | Status codes | Credential | Modes | Feature |
|---|---|---|---|---|---|---|---|
| GET | /api/v1/rules |
none | Prometheus rules JSON envelope: data.groups |
200, 401 | Yes | all, query |
none |
| GET | /api/v1/alerts |
none | Prometheus alerts JSON envelope: data.alerts |
200, 401, 503 | Yes | all, query |
none |
Both routes read a snapshot that the alert evaluator in the same process
publishes at the end of each tick. They issue no object-store request and run
no query. They serve current state, not history, and there is no
cross-process read: a process that runs no evaluator does not ask one that
does. Past transitions are in the alerts SQL table.
/api/v1/rules returns the alert rules that this process loaded, for the
authenticated tenant, in the shape that Prometheus's rules API uses.
- Its two status codes are the whole set: 200 for a resolved credential and 401 for one that does not resolve.
- There is no request parameter. The tenant comes from the credential and nothing else.
- A tenant with no loaded rules gets
{"status": "success", "data": {"groups": []}}, never the rules of another tenant, from every process, including one that runs no evaluator.
A tenant's rules render as one group named ravel-alert-rules. The Ravel
rules document is one flat rules array with no group blocks. So the name is
a constant and the group's file is the empty string. The group's interval
is the evaluation interval (--alert-eval-interval-secs) in seconds.
Each rule carries type (always alerting), name (the rule_id), query,
duration (the for delay in seconds, 0 when unset), labels,
annotations, health, and state. After a tick it also carries
lastEvaluation when health is ok or err, lastError when health is
err, and an alerts array when the tick read the tenant's alert history.
query is the rule's whole firing expression:
- For a PromQL rule, it is the query text with its threshold comparison
appended (
max by (instance) (cpu_usage) > 0.9). - For a SQL detection rule, it is the statement alone. Its condition is that the statement returned a row, and there is nothing to append.
health and state come from the last tick of this process's evaluator for
the tenant:
healthisokwhen the tick ran the rule, anderrwithlastErrorwhen the rule's query, condition or write failed, or when the tick could not read alert history or take the tenant's lease.lastEvaluationis the tick's clock reading, in RFC 3339.lastErrorfor a failed query is the message the PromQL or SQL query endpoint returns for the same failure: a PromQL parse error keeps its own text, and a storage fault reads as a fixed message such asupstream storage temporarily unavailable. A history read failure readsalert history unreadable, a lease failurealert lease unavailable, and a failed record writealert record write failed. NolastErrorcarries an object key, the tenant hash or the storage backend's own error text; the full error goes to the server log.stateisfiringwhen any of the rule's alerts is firing, elsependingwhen any is pending, elseinactive. Thealertsarray lists those alerts in the/api/v1/alertsshape.- When the last tick could not read the tenant's alert history, the first
tick included,
stateisunknownand there is noalertsarray, whilehealthiserrwithlastErroralert history unreadable. - Before this process has finished a tick for the tenant, and in a process
that runs no evaluator,
healthandstateare bothunknown, with nolastEvaluationand noalertsarray. The same holds when the tenant's last snapshot is stale (see the/api/v1/alertsstatus codes below). - On a replica whose peer held the tenant's lease on the last tick,
healthisunknownwith nolastEvaluation, andstateandalertscome from the alert history that tick read. unknownis Prometheus's own value forhealth. Forstate,unknownis outside Prometheus's three values (inactive,pending,firing).inactiveasserts that a rule is not firing, and nothing has checked.
/api/v1/alerts returns the authenticated tenant's pending and firing
alerts as of the last tick, in the shape that Prometheus's alerts API uses.
Suppressed and resolved alerts are not listed. Each alert carries labels
(with alertname set as the Alertmanager sink sets it), annotations,
state, activeAt and value.
activeAtis the transition into the current state, in RFC 3339. For a firing alert that was pending first, it is when the alert fired, where Prometheus keeps the time it went pending.valueis a string: the sample the alert's series matched with on the last tick, or for a SQL detection rule the number of rows its statement returned. It is present only when this process held the tenant's lease on that tick and ran the alert's rule without error.
Status codes:
- 200 with the tenant's alerts. A tenant with no loaded rules always gets
{"status": "success", "data": {"alerts": []}}, never the alerts of another tenant, from every process, whatever other tenants have configured and whether this process runs an evaluator. - 401 for a credential that does not resolve.
- 503 with
errorTypeunavailable, for a tenant with loaded rules only, when this process runs no evaluator (evaluation disabled), before this process has finished a tick for the tenant, and whenever the last tick could not read the tenant's alert history, until a later tick reads it. An empty list would assert that nothing is firing, and an earlier tick's list may no longer be current. - The same 503 when the tenant's last snapshot is stale: older than three
evaluation intervals, measured from the evaluator's clock reading when it
published the snapshot at the end of its tick, so a slow tick still
publishes a snapshot that is served. The evaluator is the only writer of
the snapshot, so a stopped or stuck evaluator would otherwise leave it
served unchanged. The
errorstarts withthis tenant's alert evaluator in this process has stopped publishing its stateand names the snapshot's age and the bound. Serving resumes when the evaluator publishes again.
Unauthenticated, and served in every mode, including maintain mode. The
agents health route in the query table above is also unauthenticated but is
mode- and feature-gated and needs --agents; it does not inherit this
section's rule.
| Method | Path | Request | Response | Status codes | Credential | Modes | Feature |
|---|---|---|---|---|---|---|---|
| GET | /healthz |
none | ok |
200 | No | all modes | none |
| GET | /readyz |
none | empty body | 200, 503 | No | all modes | none |
| GET | /-/healthy |
none | ok |
200 | No | all modes | none |
| GET | /-/ready |
none | empty body | 200, 503 | No | all modes | none |
| GET | /metrics |
none | Prometheus text exposition | 200 | No | all modes | none |
/healthz (and its Prometheus spelling /-/healthy) is liveness. A request
that reaches the handler proves that the server task can route. The route is
200 whenever it answers, and a store outage never makes it fail.
/readyz (and /-/ready) is readiness, the AND of four conditions:
- Startup has completed: config parsed, the object-store capability gate passed, listeners bound.
- The process is not draining.
- The store is reachable.
- No ingest shard actor has been condemned.
Only the store condition recovers on its own. The startup latch and the drain latch are one-way. A condemned ingest shard cannot recover in-process, so a 503 from that cause persists until the process is rolled.
When a shard actor is condemned depends on the signal:
- The metrics pipeline respawns a dead shard actor. It condemns only on the death that exhausts its respawn budget.
- The logs and spans pipelines never respawn, so their first shard-actor death condemns.
See the observability guide.
The readiness route issues no object-store request and takes no lock. Each condition is an atomic load, including the ingest one, which reads the condemned-shard counter. A background store probe with hysteresis supplies the store atomic. Four consecutive failed probes flip readiness to 503, and a single success recovers it.
Readiness never restarts the process. A 503 sheds traffic: Kubernetes
removes the pod from its Service endpoints. Liveness is the separate
/healthz route for that reason.
The same four probe routes, and no others, are also served on the optional
--listen-health listener. It runs on its own thread and answers even when
every main-runtime worker is busy. On that listener:
/healthzand/-/healthyalso return 503 once the main runtime's heartbeat is older than 60 s./readyzand/-/readyalso return 503 once it is older than 30 s.
/metrics is the Prometheus scrape endpoint. It is unauthenticated, so
per-tenant labels on the admission and query families are opt-in
(--metrics-tenant-labels). By default every tenant folds into a single
tenant_hash="other" series.
- Flight SQL is behind the
flight-sqlcargo feature and is a gRPC service on the gRPC listener, not an HTTP route. The published image builds that feature. - OTAP (OpenTelemetry Arrow Protocol) metrics ingest is a gRPC service that
needs both the
otapcargo feature and the--otapruntime flag. The published image builds the feature, and the flag is still required. - OTLP ingest for metrics, logs, and traces is also available over gRPC on the gRPC listener.