Skip to content

feat: manage backup retention in Klio instead of Kopia - #238

Open
gabriele-wolfox wants to merge 23 commits into
mainfrom
dev/168
Open

feat: manage backup retention in Klio instead of Kopia#238
gabriele-wolfox wants to merge 23 commits into
mainfrom
dev/168

Conversation

@gabriele-wolfox

@gabriele-wolfox gabriele-wolfox commented Sep 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Replace Kopia's snapshot-expire retention with a retention policy manager that Klio owns: Klio evaluates its own backup catalog and deletes the expired backups, rather than delegating to kopia snapshot expire.

Highlights

  • New retentionPolicy in the PluginConfiguration with a single latest criterion (keep the N most recent backups), configurable per tier and validated to be at least 1. Omitting it keeps every backup.
  • The tier2 snapshot pins are removed. The invariant that a tier1 backup is never deleted before it reaches tier2 now lives in the retention selection, read from our own catalog, so snapshot manifest IDs stay stable.
  • Retention travels to the server on the CloseBackup request and is applied per tier by the backup consumer after every backup.
  • New klio retention apply command (and ApplyRetention gRPC call) to apply the configured retention on demand (eg. without waiting for the next backup, to reclaim space quickly after tightening the policy).
  • The old klio retention set/get CLI and the Kopia retention-policy plumbing are removed.

Testing

  • Unit tests cover the pure Evaluate selection and the consumer's applyRetention (against a fake client).
  • A tier1 retention e2e keeps two of three backups, verifies the retention manager deletes the oldest, then tightens the policy to one and verifies klio retention apply leaves only the newest.
  • The tier2 retention e2e runs the same shared flow (automatic retention plus the on-demand klio retention apply) and adds tier2-only WAL retention and recovery-gate checks.

Closes #168.

@gabriele-wolfox
gabriele-wolfox force-pushed the dev/168 branch 4 times, most recently from 5972f01 to c1def0c Compare September 8, 2026 07:30
@gabriele-wolfox
gabriele-wolfox requested review from a team and jlong49 as code owners September 8, 2026 07:30
@armru
armru force-pushed the dev/168 branch 3 times, most recently from 7d6e631 to 0bab821 Compare September 14, 2026 10:41
gabriele-wolfox and others added 17 commits September 14, 2026 14:02
@gabriele-wolfox
feat: manage backup retention in Klio instead of Kopia
a0f3d65 
Replace Kopia's snapshot-expire retention with a Klio-managed policy
manager that evaluates the backup catalog and deletes the expired
backups itself. The only criterion for now is latest, which keeps the N
most recent backups; it is configured per tier in the PluginConfiguration
and must be at least 1.

Because retention no longer runs through Kopia, the tier2 snapshot pins
are gone. The invariant that a tier1 backup is never deleted before it
reaches tier2 now lives in the retention selection, read from our own
catalog, which keeps snapshot manifest IDs stable.

The tier1 and tier2 policies travel to the server on the CloseBackup
request, and the consumer applies them per tier after every backup. The
klio retention CLI and the Kopia policy plumbing are removed.

Assisted-by: Claude

Signed-off-by: Gabriele Quaresima <gabriele.quaresima@enterprisedb.com>
@gabriele-wolfox
feat(retention): apply retention on demand with a new command
67a73fa 
Add a klio retention apply command and an ApplyRetention gRPC call that
apply the configured retention immediately, without waiting for the next
backup. This frees space on demand, for example after shrinking the
policy on a nearly-full disk.

The command reads the client configuration and sends the tier1 and tier2
policies to the server, which enqueues a maintenance-only task. The
backup consumer then runs the same per-tier retention path used after a
backup, so there is a single retention implementation.

Assisted-by: Claude

Signed-off-by: Gabriele Quaresima <gabriele.quaresima@enterprisedb.com>
@armru @gabriele-wolfox
fix(config): decode the tier1 retention policy from the client config
d088904 
The operator writes the client config with the `tier1_retention` key while
the mapstructure tag said `retention`, so viper left the field nil and the
tier1 retention policy never reached the server: tier1 retention was a
no-op. Both synced copies of the struct are fixed and a decode test covers
the retention blocks.

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
@armru @gabriele-wolfox
fix(server): disable Kopia's own snapshot retention
b37cae9 
The `kopia repository create` command stores Kopia's default retention
counters in the global policy and every `kopia snapshot create` applies
the effective policy, so with the pins gone Kopia kept expiring tier1
snapshots behind Klio's back, per source, capped by its own buckets.

The server now disables Kopia retention at start, before the Kopia servers
come up: the global policy of each tier is set to keep everything, and the
per-source policies left by earlier Klio versions are reset to inherit it.

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
@armru @gabriele-wolfox
test(retention): cover applyRetention with a fake client
238a25b 
The deletion glue (catalog listing, policy evaluation, keep guard, error
joining) had no unit test. Narrow the client parameter to the two methods
it uses so a fake can drive it.

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
@armru @gabriele-wolfox
fix(retention): reject an empty or invalid cluster name in ApplyReten…
33b897a 
…tion

An empty cluster name is a wildcard for the backup listing, so a
maintenance task carrying one evaluated the policy over every cluster's
catalog and then failed on each delete, blocking the single-ack consumer
through its retry budget. The RPC now rejects an empty or invalid name and
the consumer refuses to apply retention without one.

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
@armru @gabriele-wolfox
fix(retention): retry a maintenance-only task when tier1 maintenance …
aa2ca83 
…fails

Best-effort tier1 maintenance is right after a backup, which must not fail
on it. A maintenance-only task has no backup to protect: the retention run
is the whole task, so a tier1 failure was logged once and acked as done
while the client had been told it was scheduled. It now fails the task so
the queue retries it.

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
@armru @gabriele-wolfox
fix(queue): do not purge the backup dead-letter queue on maintenance …
2b5651a 
…tasks

The purge after a successful task exists because a completed backup makes
the earlier failures moot. A maintenance-only task reuses the same handler,
so an on-demand retention run erased dead-lettered backup tasks without any
backup having succeeded.

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
@armru @gabriele-wolfox
fix(backup): delete the metadata snapshot of a backup last
b12a93d 
The snapshots of a backup were deleted in listing order. A deletion
interrupted after the metadata snapshot left the data snapshots orphaned
and invisible to the catalog, so retention could never finish the job.
Deleting the metadata last keeps a partially deleted backup listed until
the next run completes it.

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
@armru @gabriele-wolfox
fix(operator): treat a pruned retention block as no policy
43ae282 
A PluginConfiguration stored before the `latest` field existed reads back
with `latest: 0` once the old keys are pruned. The operator emitted a
retention block for it, the client rejected the config and every backup of
the cluster failed until the resource was edited. Such a block now
converts to "no policy", keeping every backup; the CRD validation still
rejects `latest` below 1 on write.

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
@armru @gabriele-wolfox
test(e2e): list only the cluster's backups in the tier2 retention fea…
37332e0 
…ture

The helper read every backup known to the Klio server, so the exact-count
assertions would fail as soon as another cluster shared the server.

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
@armru @gabriele-wolfox
chore(retention): drop leftovers of the Kopia pin based flow
ae33a8d 
Remove the consumer options only read by the deleted tier1 server refresh,
fix a comment naming a method that no longer exists, correct the upgrade
note (only `klio retention get` and `set` went away, `apply` is new) and
update AGENTS.md, which still described the pins and the tier1 unpin.

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
@armru @gabriele-wolfox
fix(retention): require a complete tier2 copy before tier1 deletes a …
8b6529b 
…backup

The tier1 retention guard considered a backup safe to delete as soon as its
metadata snapshot appeared on tier2. The relay is a single snapshot
migration with no ordering between a backup's parts, so a relay that dies
midway can leave the metadata on tier2 without the data, and the next
maintenance run would then delete the only complete copy on tier1.

The guard now lists the snapshots of both tiers and keeps a backup until
every tier1 snapshot of it (pgdata, metadata, control data, tablespaces)
has a counterpart on tier2.

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
@armru @gabriele-wolfox
fix(retention): fail the backup listing when a metadata snapshot is u…
7e94023 
…nreadable

ListBackups skipped a backup whose metadata could not be read and returned
the rest as if the catalog were complete. The catalog now drives base
backup retention and WAL cleanup, so a transient read error made them
delete one backup, or the WALs of one backup, too many. A metadata read
error now fails the listing, and the callers fail closed.

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
@armru @gabriele-wolfox
fix(retention): bind ApplyRetention to the client certificate cluster
fdc5b61 
The WAL server requires a client certificate but never bound its Common
Name to the cluster named in the request, so any certificate signed by the
CA could apply a retention policy to another cluster's backups. The host
part of the userName@hostName Common Name must now match the cluster in
the ApplyRetention request; the other calls are unchanged.

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
@armru @gabriele-wolfox
fix(retention): do not wait for tier2 on backups that skip the relay
3de63fe 
With tier2 enabled, the tier1 retention guard protected every backup that
had not reached tier2, including backups taken with tier2 backup disabled,
which never will. Those backups were never deleted from tier1.

The client now records in the backup metadata whether it asked for the
tier2 relay, and the guard lets tier1 retention delete a backup that was
never meant to reach tier2. Backups taken by older clients carry no
annotation and keep the protective behaviour.

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
@armru @gabriele-wolfox
docs(retention): document backup ordering and the tier2 guard
c72e651 
Retention orders backups by the start time taken from the instance clock,
so a clock skew after a switchover can misorder them. Spell that out, and
describe when tier 1 retention waits for tier 2.

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
@gabriele-wolfox
test(retention): verify tier1 retention and share the flow with tier2
86d9531 
Add a tier1-only e2e that mirrors the tier2 retention test: configure
a tier1 retention policy of latest:2, take three backups, and assert
the retention manager keeps exactly two and deletes the oldest, then
tighten the policy to one and verify `klio retention apply` leaves
only the newest.

Both retention features run the same tier-parameterized flow
(verifyRetentionAndOnDemandApply): the tier1 and tier2 Run methods
differ only in the tier annotation, the policy the on-demand step
edits, and the tier2-only WAL and recovery-gate checks layered on
top. Generalize the tier2 retention helpers accordingly, add a
Tier1RetentionPolicy option to the Klio plugin-configuration
template, register the new Tier1Retention feature, and update the
e2e test-structure docs.

Assisted-by: Claude

Signed-off-by: Gabriele Quaresima <gabriele.quaresima@enterprisedb.com>
@armru
test(e2e): count only the cluster's backups in the retention checks
9f47687 
The count check already decoded the cluster name of each backup but
counted every cluster's backups, while the name listing filtered by
cluster. The exact-count assertions of the shared retention flow now
look at the cluster under test only.

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
@armru
test(e2e): verify the tier1 WAL horizon follows base retention
1ae8983 
The tier1 retention feature checked that base backups are deleted but not
that the WAL horizon moved with them, which is the part of the WAL cleanup
this change touches: the first required WAL is now recomputed from Klio's
catalog after retention. After the on-demand apply leaves the newest
backup, the test waits until no tier1 WAL older than that backup's begin
WAL survives.

The tier1 WAL listing helpers move from the e2e package to the Klio
features package so both tests can use them.

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
@armru
fix(retention): let tier1 delete backups that tier2 retention already…
cdc8fee 
… removed

The tier1 guard kept every backup whose snapshots were missing on tier2,
which conflates "not relayed yet" with "relayed and deleted since". With a
tier2 policy tighter than tier1's, tier2 retention runs first and removes
the older backups; tier1 then kept those same backups, and their WALs,
forever, with only an info log that reads as transient.

The relay migrates every tier1 snapshot of the cluster at once, so a newer
backup complete on tier2 proves the relay ran after the older one existed.
A backup absent from tier2 is now deletable when such a newer backup
exists; one that is only partially on tier2 is still kept, since a relay
may be in flight or have failed midway.

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
@armru
fix(operator): make the retention latest field optional in the CRD
40564f5 
A required `latest` makes PluginConfigurations stored by earlier versions,
which carry the pruned Kopia-style keys and no `latest`, fail schema
validation on their next write, including the controller's own status
update, unless the cluster ratchets CRD validation. The field is optional
with a plain minimum of 1; an empty block converts to "no policy" as
before.

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
@armru
docs(agents): list the retention config files among the synced pairs
8b4fd29 
Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jlong49 jlong49 Awaiting requested review from jlong49 jlong49 is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Retention policies are not atomic across a backup's snapshots

2 participants

@gabriele-wolfox @armru