keep

Brief

Cloud-native, multi-core key-value / claim-check store — the loom/relay data plane and a Redis / Dragonfly replacement. Promoted from cclab-kv: the sharded engine and tiered RAM+disk persistence are unchanged; the transport is now HTTP/2 + OpenAPI (no raw TCP).

keep spec gen --lang ts|py|rust [--target <profile>] --out <dir> uses the shared in-tree generator. clients/codegen.toml pins each language's default; every output includes .openapi-codegen.json with the selected contract.

Capabilities

A promise with no gate under it is not claimed.

The RuntimeTool baseline capabilities selected by aw.toml are mandatory for this long-running service class. They do not replace Keep's product capabilities; KV, collections, durability, HA, and the relay data plane remain first-class domain roots.

Capability Index

Capability	Root WI	Notes
CLI Interface	-	mandatory baseline: binary, OpenAPI, probes, metrics, and graceful drain
Competitive KV Feature Parity	108	mandatory baseline: Redis/Dragonfly replacement breadth remains open on relay worker contract
Competitive KV Performance	126	mandatory baseline: engine ratchet and external Redis/Dragonfly comparison are not release-closed
Long-Running Stability	121	mandatory baseline: durable recovery and drain pass locally; multi-node raft remains staged
Security Hardening	-	mandatory baseline: request limits exist; auth/TLS/negative gates are still open
HTTP/2 API List	-	mandatory baseline: concise HTTP/2 route list, OpenAPI pointer, probes, and metrics
Kubernetes-Native Deployment	-	mandatory baseline: kustomize manifests, probes, drain, env config, and nonroot image
Primary Replicas	121	mandatory baseline: raft-backed primary/replica topology and failover remains staged
KV API	-	domain: scalar KV, batch, scan, locks, probes, metrics, and claim-check blobs
Collections	-	domain: hash, set, sorted-set, and list APIs
Durability	-	domain: WAL-backed durable-before-ack recovery
HA / Raft	121	domain: raft-backed HA without changing the public KV API
Relay Worker Data Plane	108	domain: claim-check/value plane paired with relay's ordered queue contract

CLI Interface

Expose Keep as a runnable long-lived binary with stable config, HTTP/2/OpenAPI, health/readiness probes, metrics, and graceful drain behavior.

• Root WI: none; this capability predates the tracker.
• Surfaces: CLI: keep - long-running HTTP/2 key-value service process.; HTTP: /openapi.json, /healthz, /readyz, /metrics - binary-served operational surface.
• Gate — behavior: cargo test -p keep --test http_api - binary-facing OpenAPI/probe/metrics and API conformance
• Source: tests/http_api.rs, src/bin/keep.rs

Work Root	Kind	WI	Gate / Evidence
keep-process-interface	epic	-	src/bin/keep.rs; tests/http_api.rs
openapi-probe-metrics-surface	epic	-	tests/http_api.rs
graceful-drain-readiness-flip	epic	-	tests/http_api.rs

Competitive KV Feature Parity

Keep covers the Redis/Dragonfly replacement breadth expected from this runtime class: scalar KV, claim-check blobs, TTL/locks, scans, collections, and the relay-facing worker data-plane integration.

• Root WI: #108
• Surfaces: HTTP: /kv/*, /hashes, /sets, /zsets, /lists, /locks - Redis/Dragonfly-style data plane over HTTP/2.; Rust API: keep::client::KvClient - in-tree HTTP client.
• Gate — behavior: cargo test -p keep --test http_api --test collections_api
  • public KV, collection, lock, TTL, and claim-check conformance
• Source: tests/http_api.rs, tests/collections_api.rs, README.md

Work Root	Kind	WI	Gate / Evidence
redis-dragonfly-api-breadth	epic	-	tests/http_api.rs; tests/collections_api.rs
http-key-value-surface	epic	-	tests/http_api.rs
hash-set-sorted-set-operations	epic	-	tests/collections_api.rs
relay-keep-worker-facing-contract	epic	108	pending integration spec

Competitive KV Performance

Keep performance claims stay tied to repeatable engine/resource gates and an external Redis/Dragonfly comparison, not anecdotal local timings.

• Root WI: #126
• Surfaces: Meter/Vat: vat.toml#meter-efficiency - isolated meter execution for performance-relevant API and durability gates.; Example: bench_compare - Redis/Dragonfly comparison harness.; Meter: external-contracts/competitor-performance/efficiency/meter-performance-gates.md
  • engine throughput and resource gate.
• Gate — efficiency: vat run meter-efficiency - meter-owned runtime evidence inside vat
• Gate — behavior: cargo test -p keep - API behavior under the performance-relevant surfaces
• Source: vat.toml, external-contracts/competitor-performance/efficiency/meter-performance-gates.md, examples/bench_compare.rs

Work Root	Kind	WI	Gate / Evidence
vat-meter-runtime-gate	epic	126	vat.toml#meter-efficiency
engine-throughput-ratchet	epic	126	external-contracts/competitor-performance/efficiency/meter-performance-gates.md
redis-dragonfly-comparison	epic	126	examples/bench_compare.rs

Long-Running Stability

Run as a long-lived data-plane service without losing durable writes across restart, without receiving traffic during drain, and with a path to raft-backed HA that preserves the public KV API.

• Root WI: #121
• Surfaces: CLI: keep - long-running WAL/snapshot-backed service process.; K8s: k8s - StatefulSet/PDB deployment shape.; Rust API: keep::raft - raft-runtime-backed HA path.
• Gate — stability: cargo test -p keep --test durability --test http_api --test raft_node - recovery, drain, probe, and raft state-machine conformance
• Source: tests/durability.rs, tests/http_api.rs, tests/raft_node.rs, .aw/tech-design/projects/keep/logic/ha-raft-sharding-roadmap.md, k8s

Work Root	Kind	WI	Gate / Evidence
wal-backed-cold-recovery	epic	-	tests/durability.rs
graceful-shutdown-and-readiness-drain	epic	-	src/bin/keep.rs; tests/http_api.rs
multi-node-http-2-raft-network	epic	121	.aw/tech-design/projects/keep/logic/ha-raft-sharding-roadmap.md

Security Hardening

Keep the long-running KV service safe by enforcing request boundaries and adding explicit negative gates for authn/z, TLS, network policy, and malformed or oversized request handling before production readiness.

• Root WI: none; this capability predates the tracker.
• Surfaces: Guard/Vat: vat.toml#guard-security - isolated guard scan with meter runtime evidence.; HTTP: keep public API - request body and data-plane boundary.; K8s: k8s - deployment boundary for future network policy and identity.
• Gate — security: vat run guard-security
  • guard-owned static/runtime evidence
• Gate — behavior: cargo test -p keep --test http_api - body-limit and public-route smoke
• Source: vat.toml, src/http/routes.rs, tests/http_api.rs

Work Root	Kind	WI	Gate / Evidence
body-limit-and-public-route-boundary	epic	-	src/http/routes.rs; tests/http_api.rs
auth-tls-network-policy-boundary	epic	-	pending guard/negative security inventory
guard-static-runtime-evidence	epic	-	vat.toml#guard-security
malformed-and-oversized-request-negative-tests	epic	-	vat.toml#guard-security

HTTP/2 API List

Publish the supported HTTP/2 API surface as a compact endpoint inventory, with probe, metrics, and OpenAPI pointers, without making OpenAPI completeness the capability definition. Every HTTP request is correlatable end to end: W3C traceparent is honored when present and a local root trace is created when absent, with the ids flowing into every request span and structured log line. Server-Timing per-response latency attribution (the shared service-http::server_timing contract) is wired into keep's HTTP stack: every response carries a Server-Timing: app;dur=<ms> baseline (#2490).

• Root WI: none; this capability predates the tracker.
• Surfaces: HTTP: /kv/*, /hashes, /sets, /zsets, /lists, /locks, /healthz, /readyz, /metrics, /openapi.json, /docs - concise HTTP/2 API list for operators and client authors.; Logs: structured stdout with per-request trace correlation — the shared service-http trace layer (service_http::trace_layer()) accepts a valid W3C version-00 traceparent (invalid input is treated as absent) and generates a fresh local root context otherwise, so every request span and log line carries trace_id/span_id/parent_span_id/trace_flags.; HTTP: Server-Timing response attribution — shared service-http::server_timing contract (Server-Timing: app;dur= per-response latency) on every response.
• Gate — behavior: cargo test -p keep --test http_api --test collections_api
  • public route list and data-plane conformance
• Source: README.md#http-surface-v1, tests/http_api.rs, tests/collections_api.rs

Work Root	Kind	WI	Gate / Evidence
http2-api-route-list	epic	-	README.md#http-surface-v1
openapi-docs-probes-and-metrics	epic	-	tests/http_api.rs
collection-route-breadth	epic	-	tests/collections_api.rs

Kubernetes-Native Deployment

Run Keep as a Kubernetes-native stateful service with PVC-backed instances, headless and ClusterIP service discovery, probe-aware readiness drain, bounded config through env/flags, and nonroot container images.

• Root WI: none; this capability predates the tracker.
• Surfaces: K8s: k8s - StatefulSet, headless/ClusterIP Services, ConfigMap, PDB, and overlays.; Container: Dockerfile and Dockerfile.release - repo-root and published-binary images.
• Gate — behavior: cargo test -p keep --test http_api - readiness, metrics, and graceful drain behavior
• Gate — stability: kubectl apply -k k8s/overlays/dev - deployment dogfood path
• Source: k8s, Dockerfile, Dockerfile.release, tests/http_api.rs

Work Root	Kind	WI	Gate / Evidence
statefulset-pvc-service-topology	epic	-	k8s
readiness-drain-and-probe-contract	epic	-	tests/http_api.rs
nonroot-container-image-path	epic	-	Dockerfile; Dockerfile.release

Primary Replicas

Support a primary/replica HA topology where one write-owning primary is backed by durable replicas and failover preserves the public KV API. The capability is tracked separately from the generic HA domain root so the profile explicitly names the replica requirement.

• Root WI: #121
• Surfaces: Rust API: keep::raft - raft-runtime-backed state machine for primary/replica convergence.; K8s: StatefulSet - stable network identity and PVC-backed replica pods.; HTTP: future raft network - primary write ownership with replica failover.
• Gate — stability: cargo test -p keep --test raft_node - current raft state-machine conformance
• Gate — behavior: multi-node HTTP/2 raft network - staged failover gate
• Source: tests/raft_node.rs, .aw/tech-design/projects/keep/logic/ha-raft-sharding-roadmap.md, k8s

Work Root	Kind	WI	Gate / Evidence
single-node-raftcore-state-machine	epic	121	tests/raft_node.rs
primary-replica-topology-contract	epic	121	.aw/tech-design/projects/keep/logic/ha-raft-sharding-roadmap.md
multi-node-replica-failover	epic	121	.aw/tech-design/projects/keep/logic/ha-raft-sharding-roadmap.md; k8s

KV API

Expose a cloud-native key-value and claim-check store over HTTP/2 + OpenAPI, including scalar get/set/delete, batches, scans, locks, probes, metrics, and opaque blob roundtrips.

• Root WI: none; this capability predates the tracker.
• Surfaces: HTTP: /kv/*, /healthz, /readyz, /metrics, /openapi.json - public service API.; Rust API: keep::client::KvClient - in-tree HTTP client.
• Gate — behavior: cargo test -p keep --test http_api - public HTTP API conformance
• Source: tests/http_api.rs

Work Root	Kind	WI	Gate / Evidence
http-key-value-surface	epic	-	tests/http_api.rs
claim-check-blob-roundtrip	epic	-	tests/http_api.rs

Collections

Provide Redis-like hash, set, sorted-set, and list operations on the same durable engine and HTTP surface.

• Root WI: none; this capability predates the tracker.
• Surfaces: HTTP: /hashes, /sets, /zsets, /lists - collection APIs.
• Gate — behavior: cargo test -p keep --test collections_api - collection operation conformance
• Source: tests/collections_api.rs

Work Root	Kind	WI	Gate / Evidence
hash-set-sorted-set-operations	epic	-	tests/collections_api.rs
list-push-pop-blocking-pop-operations	epic	-	tests/collections_api.rs

Durability

Persist mutations before acknowledgement and recover committed state after a cold restart.

• Root WI: none; this capability predates the tracker.
• Surfaces: Engine: KvEngine + persistence - WAL and snapshot-backed state.; HTTP: mutation APIs - durable-before-ack public writes.
• Gate — stability: cargo test -p keep --test durability - cold recovery conformance
• Source: tests/durability.rs
• Evidence: tests/durability.rs

HA / Raft

Move keep from independent StatefulSet shards toward raft-backed HA without changing the public KV API.

• Root WI: #121
• Surfaces: Rust API: keep::raft - raft-runtime-backed state machine.; K8s: StatefulSet - PVC-backed instances.
• Gate — stability: cargo test -p keep --test raft_node - raft state-machine conformance
• Source: tests/raft_node.rs, .aw/tech-design/projects/keep/logic/ha-raft-sharding-roadmap.md

Work Root	Kind	WI	Gate / Evidence
single-node-raftcore-state-machine	epic	121	tests/raft_node.rs
multi-node-http-2-raft-network	epic	121	.aw/tech-design/projects/keep/logic/ha-raft-sharding-roadmap.md

Relay Worker Data Plane

Serve as the claim-check/value data plane paired with relay's ordered queue and worker contract.

• Root WI: #108
• Surfaces: HTTP: keep OpenAPI + relay OpenAPI - worker payload and lease integration contract.
• Gate — behavior: future relay+keep integration gate - worker-facing contract closure
• Source: README.md, apps/relay/tests/worker_loop.rs
• Evidence: pending integration spec

Architecture

Crate	Path	Notes
keep	crates/keep/	Top-level facade library (keep::*), CLI/server binary (src/bin/keep.rs), integration tests, benchmarks, and examples.
keep-core	crates/keep-core/	Core value types (KvValue, SetOptions, JSON conversion), error model (KvError), shard cluster topology, Prometheus metrics, and TLS helpers.
keep-engine	crates/keep-engine/	Sharded multi-core KV engine (KvEngine), durable WAL + snapshot persistence (persistence/), backup/restore, and optional raft state machine.
keep-http	crates/keep-http/	Axum over hyper HTTP/1.1 + HTTP/2 cleartext server, routes, handlers, request/response models, and OpenAPI spec generation (/openapi.json).
keep-operator	crates/keep-operator/	Kubernetes KeepCluster CRD, manifest renderer, and reconciliation loop.
keep-client	crates/keep-client/	In-tree async HTTP/2 client (KvClient) for scalar KV, batch, lock, and claim-check operations.

The engine and persistence layers are transport-agnostic. Polyglot workers integrate against the generated OpenAPI document — there is no language-specific client to ship (relay+keep worker contract, #108).

HTTP surface (v1)

Method	Path	Op
GET/PUT/DELETE/HEAD	/kv/{key}	get / set / delete / exists
POST	/kv/{key}/incr	atomic incr/decr (signed delta)
POST	/kv/{key}/cas	compare-and-swap
POST	/kv/{key}/setnx	set-if-absent
POST	/kv:mget :mset :mdel	batch
GET	/kv?prefix=&limit=	scan
POST/DELETE/PATCH	/locks/{key}	acquire / release / extend lease
POST/GET	/lists/{key} + /{lpush,rpush,lpop,rpop,blpop,brpop} /length	list push/pop/blocking-pop/range/len
POST/GET/DELETE	/hashes/{key} + /length /mget /incr /fields/{field}	hash ops
POST/GET/DELETE	/sets/{key} + /length /members/{m}	set ops
POST/GET/DELETE	/zsets/{key} + /length /incr /members/{m}/{score,rank}	sorted-set ops
POST/GET	/kv/{key}/{expire,ttl,persist,getex}	TTL / expiry on any key
GET	/healthz /readyz /metrics /info /openapi.json /docs	admin / probes

Values. Structured values travel as native JSON (application/json, body {"value": <json>, "ttl_ms": <opt>}). Opaque blobs (claim-check payloads) travel as raw bytes (application/octet-stream, TTL via ?ttl_ms=) and never round-trip through JSON — GET returns them verbatim as octet-stream.

Durability. All mutations are WAL-backed and durable-before-ack — scalars (/kv, incr/cas/setnx, :mset/:mdel), collections (hash / set / sorted-set / list push+pop), and TTL ops (expire/persist). A write returns 200 only after its op is fsynced (group-committed); committed state survives a cold recovery (see tests/durability.rs).

Blocking pops. BLPOP/BRPOP long-poll up to timeout_ms (capped at 60 s) for an element, waking immediately on a concurrent push via a per-key notifier.

Run

cargo run -p keep --bin keep            # listens on 127.0.0.1:7117, ./data
KEEP_HOST=0.0.0.0 KEEP_PORT=7117 keep   # all config via KEEP_* env or --flags

Config: --host/--port/--shards/--data-dir/--disable-persistence/--fsync-ms/ --snapshot-secs/--snapshot-ops/--body-limit/--grace-secs, each with a KEEP_* env fallback. RUST_LOG overrides --log-level.

Kubernetes

StatefulSet (each pod owns a PVC-backed disk tier) + headless & ClusterIP Services + ConfigMap + PDB, distroless nonroot image. SIGTERM flips /readyz to 503, drains the grace window, then flushes the WAL.

kubectl apply -k k8s/overlays/dev      # or staging / prod

Images: Dockerfile (from-source, build context = repo root) and Dockerfile.release (published binary into distroless).

Status / roadmap

• ✅ HTTP/2 + OpenAPI surface, k8s-native (probes, SIGTERM drain, env config) — #114
• ✅ durable-before-ack WAL (no-drop, group commit) — #114
• ✅ perf-gate via meter: engine throughput ratchet + server resource gate — see meter-performance-gates.md (#126). Competitor comparison (vs Redis/Dragonfly) is the separate one-off examples/bench_compare.rs.
• ☐ worker-facing OpenAPI contract finalized with relay — #108
• ◑ HA — phase A (sharded scale-out + /cluster) done; phase C raft via raft-runtime integrated (--features raft, engine-backed state machine, HTTP write propose path, proven by tests/raft_node.rs and the raft HTTP API test) — multi-node failover remains staged; see .aw/tech-design/projects/keep/logic/ha-raft-sharding-roadmap.md (#121)
• ✅ queuekit and queue-era ion feature consumers now use keep::client instead of the retired raw-TCP cclab-kv client; remaining cleanup is deduping legacy cclab-kv copies.