Skip to content
This repository was archived by the owner on Nov 8, 2026. It is now read-only.

Latest commit

 

History

History
342 lines (283 loc) · 20.8 KB

reliability.md

File metadata and controls

342 lines (283 loc) · 20.8 KB

Reliability and recovery

Production incident qualification

Public qualification also exercises commit/response-loss boundaries, worker SIGKILL before and after source-offset settlement, concurrent single-record histories through two server processes, and slow-store saturation with bounded healthy-store deadlines and cancellation cleanup. Nightly qualification repeats twelve fault cycles during a two-hour workload, including simultaneous Kafka and OpenSearch outages. This long run is scheduled or explicitly requested for a special qualification; routine changes and releases do not wait for it. Release binaries and images require the shorter public production checks.

Sink reliability gate aggregates every PR check and fails if any prerequisite fails, is cancelled or is skipped. Repository rules must require this status.

Readiness requests respect their own deadline even if a dependency probe is still waiting on network work. Concurrent HTTP and gRPC health checks share at most one outstanding probe per dependency; a completed result is not cached.

Every server PR runs the immutable public suite against the candidate executable, with race detection and real Elasticsearch plus OpenSearch 3.8/2.17. The suite's incident contracts cover the production failures behind PRs 37, 38, 40 and 41: bounded hot-key work, applied/visible isolation, per-caller budgets, Replace conflicts, independent document completion, dataset refresh waits and queued cancellation.

An independent Go state model checks mixed operations through serial RPCs and several batching configurations, then through all seven configured storage routes in release qualification. Suite CI requires matching assertion failures on six immutable pre-fix server commits, so a passing regression must also have evidence that it detects the historical bug. Empty or skipped conformance runs fail qualification. Release artifacts remain blocked on the pinned public suite; nightly runs add varying state-machine seeds before the existing two-hour fault workload. Keep the PR, release and nightly suite pins synchronized.

These checks establish the listed contracts, not SQLite-level certification. The suite records remaining Sink-specific gaps, including broader compound failures, arbitrary concurrent histories and generated malformed replies. Storage fault injection qualifies Sink's acknowledgements, retry, isolation and recovery behavior; it does not certify the database's own election or backup implementation. Repository rules must require the CI statuses to enforce a pre-merge gate.

Delivery and recovery semantics

Sink provides at-least-once asynchronous delivery and per-record conditional updates. Business idempotence belongs to the application. Sink does not deduplicate logical mutations. Client retries, lost backend acknowledgements, worker restarts, and DLQ replay can repeat business effects. Use a business operation ID with atomic check-and-apply logic when repeated effects are unsafe.

Supported MongoDB native writes and record writes share the same revision protocol: business changes and fresh revision metadata commit atomically per document. A concurrent Merge retries an invalidated snapshot. This requires all writers to use the same metadata field and excludes direct database writes and administrative restore/rename paths. See revision-protected native mutations for supported commands, pre-execution rejection, and remaining native semantics. Revision conflict detection does not deduplicate business effects or make native retries safe.

Returned-document chains execute in ordered rounds: independent direct Puts commit before a round starts conditional reads, without concurrent unbudgeted snapshots. Each returned operation retains its own committed document. Scan has separate byte, request and per-store sublimits in addition to global admission. Defaults reserve half the configured capacity for ordinary work (count limits round up to one for a total limit of one). Two default BSON scans fit the 128 MiB scan budget; a third is rejected while ordinary Read/Merge can still fit. Raising scan sublimits to total limits relinquishes that reservation. MongoDB Read cursor cleanup uses a separate five-second context after caller cancellation; cleanup is best effort if the backend itself is unreachable.

Interpret outcomes

Outcome Meaning and action
ACCEPTED Kafka acknowledged the mutation. It may still be pending, applied, or quarantined. It is not a database commit receipt.
APPLIED / visible completion The backend acknowledged the requested operation/visibility. Durability depends on the configured backend and cluster.
RPC cancellation, deadline, transport failure Some effects may already have happened. Reconcile the business state before retrying operations without idempotence.
Admission rejection Core execution capacity was unavailable before execution. Back off and retry with jitter.
Per-operation permanent failure Inspect and correct the operation. Retrying identical invalid input does not repair it.
Per-operation temporary failure Retry only unresolved operations, respecting business idempotence and ordering.

Synchronous Puts and Merges for one address may share a final commit and its revision. Their intermediate documents do not create independent backend writes or change events. A failed commit leaves every operation in that run unresolved; see record operation folding before relying on per-write effects.

There is no persistent per-mutation status API. Applications that need end-to-end reconciliation should keep their own accepted-operation ledger, compare business IDs against stored data, and reconcile source lag and DLQ. An empty source lag alone does not mean all accepted mutations succeeded: some may be in DLQ.

Failure handling

Temporary backend failures are retried beyond max_retry_attempts; that setting limits attempts in one processing round. Unresolved records remain in Kafka. Workers commit only resolved contiguous partition prefixes. DLQ or commit failure retains the affected source records for replay. Permanent failure of one record does not suppress subsequent valid operations for the same address; temporary failure does preserve the same-address barrier.

Only confirmed per-record permanent errors belong in DLQ. Whole-request search errors (including authentication/authorization failures), index blocks, unknown item errors and incomplete replies retain queued work. Unknown/internal/missing failure details also preserve the same-record barrier. Retrying these errors does not imply that they will repair themselves: permission and index settings may need operator action. Search mapping rejections and MongoDB document validation, immutable-field and BSON-size rejections remain explicit record failures. A malformed read reply must not become a not-found result, and an error embedded in a bulk delete result must not become an applied acknowledgement.

Search multi-get and bulk replies must identify each document in request order; bulk replies must also match the submitted action type. Missing index names or mismatched IDs/actions make the whole reply retryable, including entries that otherwise report success, absence or a permanent failure. Index aliases can return concrete index names, so those names are required but are not compared to the requested alias.

Search requests never follow HTTP redirects. Record operations return a backend failure for a redirect; native Execute returns the original HTTP status and body. Native requests reject Idempotency-Key and X-Idempotency-Key because they would enable hidden transport retries after a lost mutation acknowledgement. Reconcile business state before explicitly retrying an ambiguous mutation.

Search response byte limits trigger read splitting without retrying the same oversized batch on other endpoints. Byte limits and caller cancellation do not mark an endpoint unhealthy. Transport failures and temporary HTTP errors still cool down the affected endpoint and allow record reads and managed Query, Count and Scan requests to fail over while their request context remains active. Native Execute commands retain a single attempt because they may mutate data or manage backend cursors.

Processing defaults to 20 seconds. A pending rebalance cancels backend work and allows at most five additional seconds for DLQ/offset settlement before releasing rebalance callbacks. This improves bounded handover but is not storage fencing. Unknown in-flight writes can complete; adapters must honor cancellation, and business operations must tolerate replay. Keep one ordered asynchronous path for order-sensitive records. Concurrent sync writes, multiple producers, and late DLQ replay need application-level version/order checks.

Kafka keys are the complete canonical record URI for every Store adapter. Deploy this format in new topics with matching publishers and Workers; old mutation messages are not supported by the new cluster. See record addresses.

After processing and offset settlement stop, worker group departure uses shutdown_timeout. If Kafka does not acknowledge departure within that window, Sink cancels the Kafka client's internal network work before closing it. Shutdown does not acknowledge additional source records; a replacement consumer recovers unresolved offsets. SIGKILL recovery also depends on Kafka's session timeout and rebalance window, so process startup alone does not prove consumption has resumed.

Source retention is still a finite recovery window. The default is 72 hours; set it longer than outage detection, repair, and backlog drain time combined. DLQ defaults to 30 days. Source offset gaps fail visibly instead of silently resetting. If a gap occurs, stop the affected path, preserve its positions, recover business state from the producer ledger/backups, choose replacement positions explicitly, and restart the worker. Do not reset to latest merely to make lag or health green.

Inspect and replay dead letters

Use the existing store configuration. Inspection joins no consumer group and changes no offsets. It reports positions, payload hashes and error/source headers; document payloads are not printed. Select a concrete range of 1–100 consecutive records, at most 64 MiB. Missing/expired offsets fail the command.

sink dlq inspect --config /etc/sink/worker.yaml --store-config /etc/sink/store.yaml --store primary \
  --partition 0 --offset 12 --count 3 > dlq-inspection.jsonl

Repair the cause, review the selected records' business intent, and preserve the inspection output in the incident record. Replay validates the entire selection's envelope/store before republishing, but publishing the selection is not atomic. It does not remove DLQ records or advance the worker's group offsets:

sink dlq replay --config /etc/sink/worker.yaml --store-config /etc/sink/store.yaml --store primary \
  --partition 0 --offset 12 --count 3 > dlq-replay.jsonl

Replay uses the complete canonical record URI as its Kafka key for every Store. This keeps replayed and newly accepted operations for the same record on the same partition when its partition count is unchanged. Replay does not recover an operation's original position in time. Replay reports accepted or failed_or_unknown for each selected position and returns failure if any publication fails. If the command or output file is interrupted, outcomes may be unknown. Never repeatedly replay a whole range to test whether the error disappeared. Reconcile its business IDs and store a resolution record containing source/DLQ positions, hash, replay time, operator, and observed business result. Resolve only after verifying persisted state. Replayed older operations can overwrite newer state unless the business rule checks its version. Malformed or wrong-store envelopes require manual correction through the normal API rather than the replay command.

Durability baseline

For production Kafka, start with RF=3, minimum ISR=2, all-ISR producer acknowledgements, clean leader election, and replicas distributed across failure domains. Loss of sufficient ISR must reject writes rather than weaken durability. Sink reconciles the Topic policy, but rack placement, broker storage, replication fetch limits, cluster permissions, and backup/recovery remain deployment work. Do not configure servers and workers with conflicting policies.

Sink's MongoDB client uses w=majority and journal=true, with bounded server selection and request deadlines. Before upgrade, verify the replica set and journaling configuration, election behavior, and capacity. A write-concern timeout does not roll back an already executed mutation.

For Elasticsearch/OpenSearch, verify index replicas across failure domains, index.translog.durability=request, shard health, disk watermarks, and snapshot restore. Sink does not automatically initialize index settings; applications can manage them explicitly through native Execute. Confirm acknowledgement and visibility semantics against the cluster actually being deployed.

Do not change partition counts online. Pause every publisher, drain and settle old work, record source/DLQ positions, create or explicitly migrate the target Topic, update all clients together, and resume only after verifying routing. Changing only a Sink config is refused to protect ordering.

Capacity, isolation and health

Gateway, Engine and Worker use process memory watermarks to admit new work. Admitted work follows its caller cancellation and existing execution boundaries; there are no forwarded memory grants or distributed usage ledgers. Engine method queues retain their existing operation and input-byte limits. Streaming calls submit one microbatch at a time and release delivered result documents before submitting the next. Worker processing is unchanged.

Each Read or returned Write result must fit its local message ceiling. Returned write candidates are checked before their own commit. Scalar Delete bounds its aggregate result envelopes before dispatch. Failure messages remain valid UTF-8, retain their codes and retryability, and are shortened to fit their local response boundary. Gateway checks each forwarded frame and propagates gRPC status details.

Write/Delete dispatchers preserve record dependencies across batches. Kafka producer buffers, backend transport bounds and Lua sandbox limits remain local. Capacity planning includes waiting queues, active microbatches, transport and driver buffers, VM working sets, concurrency and replica count. Logical message limits are not process RSS limits.

Repeated read keys produce separate streamed results without sharing a cross-Store allowance. Search reads split an oversized backend response before treating an individual document as oversized. Lua output traversal bounds alias expansion, depth and node count before constructing Go output values. The current embedded Lua VM does not provide a strict execution heap quota. Reviewed scripts and container memory limits are still required; independent Engine/Worker and Store deployments limit the impact of a process OOM. Untrusted-script isolation requires a VM with an enforced allocation quota or a separate execution process; this is not claimed by the current implementation.

Dependency failures do not prevent unrelated Stores starting. Kafka acceptance and consumption remain gated until their Topic policy is established. gRPC dependency health begins NOT_SERVING; the default health service describes the serving process. HTTP health runs independently of Prometheus; use /livez on the health port for liveness. Gateway and Engine /readyz report process readiness; Engine capability checks use /readyz?service=sink.storage.<store> or sink.kafka.<store>. Worker /readyz checks its dependencies and consumer; sink.worker.<store> selects the consumer. Gateway does not proxy dependency health. Readiness failure during an outage should not trigger liveness restart loops.

Storage capability readiness probes backend connectivity. MongoDB uses a Ping, which does not exercise majority/journal acknowledgement. A successful probe is not proof of write availability, spare capacity or a latency SLO. Qualify the required acknowledgement mode with actual traffic and reconciliation, including elections and majority loss; monitor those outcomes alongside health probes.

Protect gRPC and metrics with an authenticated TLS ingress/service mesh and network policy. Restrict who can submit Lua and which storage credentials Sink uses. The record URI is a routing address, not authorization checks. Validate these deployment controls before exposing Sink.

Monitoring and acceptance

Use example alert rules and configure Kafka exporter/group monitoring for each source and DLQ. Track source lag/oldest age, DLQ resolution lag, last committed progress, fetch/commit recovery, admission rejection, RSS, goroutines and P99 latency. Internal pending metrics cover the last fetched batch only, not unpolled backlog. End-to-end business latency needs the application ledger; the internal delivery histogram also includes quarantine.

Release evidence should include the following cases, with business outcomes and resource usage recorded rather than only API success counts:

Layer Required checks
Every change Race tests, cancellation/all-caller cancellation, CREATE replay continuation, permanent size errors, temporary outage beyond retry budget, DLQ failure without source commit, partition prefix progress, rebalance cancellation, expanded Lua/read output bounds, codec fuzz.
Scheduled or explicitly requested sustained testing Mixed sync/async and cross-store traffic; slow backends, saturation, worker/broker restarts, lag recovery; capture RSS, goroutines, latency, DLQ and business state. Use product retry defaults.
Dependency failure qualification Active-consumer SIGKILL, unavailable/blocked storage, network failures, ambiguous or malformed acknowledgements, offset retention gaps and recovery; verify Sink's outcomes and retained records.

Record the exact server/suite revisions, configuration, fault timeline, accepted business IDs, final reconciliation, maximum resource usage and backlog drain time. Define SLO, RPO and RTO from business requirements and validate them with these measurements. Passing local/fake-broker tests alone is not production reliability certification.

Write conflicts and unique constraints

Final record-write conflicts use WRITE_STATUS_PRECONDITION_FAILED. The Go SDK exposes this as WritePreconditionFailed. Inspect the failure code and retryability to distinguish the cause:

Cause Failure code Retryable
Create/Replace condition does not match PRECONDITION_FAILED false
MongoDB unique constraint rejects the document PRECONDITION_FAILED false
Internal revision or _id race exhausts retries CONFLICT true

A unique constraint failure preserves the MongoDB error, including the index name. It does not imply that the addressed record exists. Only a positively identified _id insertion race is retried as an Upsert/CAS race; secondary unique constraints do not improve by rebasing the same document. Workers treat these permanent failures as completed rejections so corrected queued writes can proceed. Transport and write-concern failures remain retryable dependency failures because they do not establish a definite rejected write.

This status describes the final outcome of synchronous completion. An asynchronous ACCEPTED response cannot include conflicts discovered later by a worker.

Search Bulk and Replace conflicts

Search compacts valid JSON before writing Bulk NDJSON, retaining number and string semantics and keeping each source on one line. A formatted document therefore cannot break a neighboring operation's Bulk framing.

RecordExists (Replace) uses a snapshot and CAS to avoid recreating a deleted record. An internal revision race is retried up to three attempts, rereading existence each time. A missing record is a permanent precondition failure; exhausted version races are retryable CONFLICT failures. Explicit revision conditions retain their original failure semantics. Successful sibling writes and transport failures with unknown commit outcomes are never replayed by this retry loop.

Oversized worker execution batches

Workers split a batch when the synchronous core rejects the entire request with RESOURCE_EXHAUSTED before execution. Splitting keeps same-record ordering and allows individually valid records to progress within a smaller execution budget. Ambiguous transport failures and per-operation failures do not trigger this resubmission. A single record that cannot fit still requires corrected input or an appropriate execution budget; it remains retryable and is not discarded.