Design doc: embed Litestream replication for sqlite-rest #121

Copilot · 2026-01-14T07:11:41Z

The issue requests a design for embedding Litestream’s Go library to add replication/restore to sqlite-rest without altering the REST API.

Design scope: Opt-in Litestream replication for the SQLite DB; restore before serve; continuous WAL/snapshot replication; logging and Prometheus metrics; fail-fast defaults (no degraded mode) and divergence handled by failing fast with operator-driven force-restore if needed.
Configuration surface: Prefer a Litestream YAML config via --replication-config (minimal sqlite-rest flags: enable/config path); detailed tuning (snapshot/retention/replicas) delegated to Litestream config; least-privilege S3 IAM notes; multi-replica refers to multiple remote destinations for the same DB, with initial scope single.
Lifecycle & failure handling: Restore pre-open, start replicator alongside HTTP/metrics/pprof, monitor error channels and cancel the process on replication errors/divergence (fail-fast).
Testing plan (future work): Flag parsing, file-backed replication integration, restore flow, and metrics assertions.

Example lifecycle sketch:

restoreIfNeeded(ctx, dbPath, restoreURL, restoreOpts)
db := openDB(...)
replicator := newReplicator(dbPath, replicaURL, tuneOpts)
go replicator.Start(ctx)
go metricsServer.Start(ctx)
go pprofServer.Start(ctx)
server.Start(ctx.Done())
// On replicator error: log/metric and cancel ctx to fail fast

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

bcho

@copilot

bcho · 2026-01-14T07:26:24Z

docs/litestream-replication-design.md

+
+### High-level flow
+
+1. **Configuration** (new `ReplicationOptions`):


change to consume litestream configuration file here to minimal the deltas.

Updated to prefer a Litestream YAML config via --replication-config so sqlite-rest changes stay minimal. (f01868e)

bcho · 2026-01-14T07:27:32Z

docs/litestream-replication-design.md

+   - `--replication-restore-lag` (duration, default `0` meaning no lag allowed; used during startup restore decisions to tolerate a small amount of staleness between the local DB and the replica before forcing a restore).
+   - Env var mirrors for container use (e.g., `SQLITEREST_REPLICATION_ENABLED`, etc.).
+
+2. **Restore before serving**:


describe the behavior of diverging with remote backup

Added divergence behavior: default fail-fast on lineage mismatch with remote, operator can force-restore or allow degraded if they accept overwrite risk. (f01868e)

bcho · 2026-01-14T07:29:20Z

docs/litestream-replication-design.md

+
+## Open questions
+
+- Should we expose multiple replicas at launch or keep single-replica until requested?


multiple replicas of what? Please clarify

Clarified this means multiple remote destinations for the same SQLite DB via Litestream; initial scope remains single replica. (f01868e)

remove this question as we have the answer

Removed the answered question about multiple replicas; doc now just states initial scope is single replica with Litestream config controlling replicas. (d41a7dd)

Removed the answered question; doc reflects single-replica initial scope with Litestream config controlling destinations. (d41a7dd)

bcho · 2026-01-14T07:29:47Z

docs/litestream-replication-design.md

+## Open questions
+
+- Should we expose multiple replicas at launch or keep single-replica until requested?
+- How strict should startup be when replication is enabled but the remote is unreachable? **Recommendation:** fail fast by default to avoid running without configured durability, with an explicit `--replication-allow-degraded` escape hatch if operators need to accept the data-loss risk.


agree with this decision, but we should refine the cli flag experience.

Flag UX is now minimal: enable + config path, optional restore-from/allow-degraded; all other tuning stays in the Litestream config. (f01868e)

remove this question

Removed the resolved question; CLI UX stays minimal via enable + config path, optional restore-from. (d41a7dd)

Removed the resolved question per feedback. (d41a7dd)

bcho · 2026-01-14T07:30:15Z

docs/litestream-replication-design.md

+
+- Should we expose multiple replicas at launch or keep single-replica until requested?
+- How strict should startup be when replication is enabled but the remote is unreachable? **Recommendation:** fail fast by default to avoid running without configured durability, with an explicit `--replication-allow-degraded` escape hatch if operators need to accept the data-loss risk.
+- What are the sensible defaults for snapshot/retention to balance durability and cost?


defaults to disable this feature, delegate the detailed settings to litestream config

Documented that replication stays disabled by default and detailed settings are delegated to the Litestream config file. (f01868e)

remove this question

Removed the answered question; defaults remain disabled with detailed settings delegated to the Litestream config. (d41a7dd)

Removed the remaining question; open questions now only cover sensible defaults for snapshot/retention. (d41a7dd)

Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

bcho

@copilot

docs/litestream-replication-design.md

bcho · 2026-01-14T07:38:20Z

docs/litestream-replication-design.md

+
+## Open questions
+
+- Should we expose multiple replicas at launch or keep single-replica until requested?


remove this question as we have the answer

bcho · 2026-01-14T07:38:25Z

docs/litestream-replication-design.md

+## Open questions
+
+- Should we expose multiple replicas at launch or keep single-replica until requested?
+- How strict should startup be when replication is enabled but the remote is unreachable? **Recommendation:** fail fast by default to avoid running without configured durability, with an explicit `--replication-allow-degraded` escape hatch if operators need to accept the data-loss risk.


remove this question

bcho · 2026-01-14T07:38:34Z

docs/litestream-replication-design.md

+
+- Should we expose multiple replicas at launch or keep single-replica until requested?
+- How strict should startup be when replication is enabled but the remote is unreachable? **Recommendation:** fail fast by default to avoid running without configured durability, with an explicit `--replication-allow-degraded` escape hatch if operators need to accept the data-loss risk.
+- What are the sensible defaults for snapshot/retention to balance durability and cost?


remove this question

Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

bcho

@copilot

bcho · 2026-01-14T07:57:19Z

docs/litestream-replication-design.md

+1. **Configuration** (new `ReplicationOptions`):
+   - `--replication-enabled` (bool, default false).
+   - `--replication-config` (string, path to Litestream YAML config; preferred path to keep sqlite-rest changes minimal and delegate detailed tuning like snapshot/retention/replicas to Litestream).
+   - `--replication-restore-from` (optional override to restore from a different replica URL; if omitted, use the primary replica from the Litestream config).


remove --replication-restore-from flag

Removed the --replication-restore-from flag; minimal flags are now enable + config path. (61a3a12)

bcho · 2026-01-14T07:57:29Z

docs/litestream-replication-design.md

+   - `--replication-config` (string, path to Litestream YAML config; preferred path to keep sqlite-rest changes minimal and delegate detailed tuning like snapshot/retention/replicas to Litestream).
+   - `--replication-restore-from` (optional override to restore from a different replica URL; if omitted, use the primary replica from the Litestream config).
+   - Env var mirrors for container use (e.g., `SQLITEREST_REPLICATION_ENABLED`, `SQLITEREST_REPLICATION_CONFIG`, etc.).
+   - Recommended CLI UX: keep flags minimal (`--replication-enabled`, `--replication-config`, optional `--replication-restore-from`) and leave all other Litestream knobs to the config file.


drop this line

Dropped the UX line tied to the removed flag; doc reflects only enable + config path. (61a3a12)

Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

Initial plan

b057512

Copilot AI assigned Copilot and bcho Jan 14, 2026

Copilot started work on behalf of bcho January 14, 2026 07:12 View session

Copilot AI and others added 9 commits January 14, 2026 07:15

docs: add litestream replication design

e79df89

Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

docs: clarify replication startup default

913d09d

Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

docs: refine multi-replica config note

bbb612e

Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

docs: align replica flag naming

322f752

Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

docs: clarify replica flag scope and error handling

46325d5

Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

docs: add go syntax hint to lifecycle sketch

f16d065

Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

docs: expand restore config and error handling

8aaf3ae

Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

docs: note s3 perms and rename degraded flag

229cb9e

Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

docs: clarify restore lag and degraded handling

6afce55

Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Design document for embedding litestream for replication~~ Design doc: embed Litestream replication for sqlite-rest Jan 14, 2026

Copilot AI requested a review from bcho January 14, 2026 07:22

Copilot finished work on behalf of bcho January 14, 2026 07:22

bcho requested changes Jan 14, 2026

View reviewed changes

Copilot started work on behalf of bcho January 14, 2026 07:31 View session

docs: shift to litestream config and clarify divergence

f01868e

Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

Copilot AI requested a review from bcho January 14, 2026 07:34

Copilot finished work on behalf of bcho January 14, 2026 07:34

bcho reviewed Jan 14, 2026

View reviewed changes

Copilot started work on behalf of bcho January 14, 2026 07:39 View session

docs: simplify flags and fail-fast defaults

d41a7dd

Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

Copilot AI requested a review from bcho January 14, 2026 07:41

Copilot finished work on behalf of bcho January 14, 2026 07:41

bcho requested changes Jan 14, 2026

View reviewed changes

Copilot started work on behalf of bcho January 14, 2026 07:58 View session

docs: remove restore-from flag and tighten ux

61a3a12

Co-authored-by: bcho <1975118+bcho@users.noreply.github.com>

Copilot AI requested a review from bcho January 14, 2026 07:59

Copilot finished work on behalf of bcho January 14, 2026 07:59


		### High-level flow

		1. Configuration (new `ReplicationOptions`):


		## Open questions

		- Should we expose multiple replicas at launch or keep single-replica until requested?

Design doc: embed Litestream replication for sqlite-rest #121

Are you sure you want to change the base?

Design doc: embed Litestream replication for sqlite-rest #121

Uh oh!

Conversation

Copilot AI commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bcho left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

bcho left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bcho left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 14, 2026

Copilot AI commented Jan 14, 2026 •

edited

Loading