DA261 S3 backup end-to-end against MicroCeph by delgod · Pull Request #60 · canonical/valkey-operator

delgod · 2026-05-13T14:54:28Z

Summary

Adds an end-to-end integration test suite for the S3 backup feature, backed by a MicroCeph-based S3 fixture. Exercises the any-unit guarantee by running create-backup on three different units and list-backups on a fourth, then verifying the resulting objects in the bucket.

Depends on

#59 must be merged first — this PR exercises actions and the relation surface introduced there.

What's in the PR

tests/integration/backup/__init__.py — package marker
tests/integration/backup/conftest.py:
- Module-scoped microceph fixture: snap install microceph, bootstrap, disk add loop,4G,3 --wipe, enable rgw with --ssl-port=445 --ssl-certificate=auto, create a test user, generate a unique bucket name, expose the rgw self-signed cert
- s3_bucket fixture: boto3 Bucket resource (creates the bucket eagerly, verify=False for the self-signed endpoint)
- s3_secret_content: access/secret-key dict for the Juju secret used by s3-integrator
tests/integration/backup/test_s3_backup.py:
- Deploys 3 valkey units + s3-integrator on Ubuntu 24.04 via jubilant
- Provisions an s3-creds Juju secret, configures the integrator (bucket, endpoint, region, path, path-style URIs), waits for active/idle
- Integrates valkey ↔ s3-integrator
- Runs create-backup on valkey/0 — asserts a backup-id matching %Y-%m-%dT%H:%M:%SZ
- Runs create-backup on valkey/1 — asserts a distinct id
- Runs list-backups on valkey/2 — asserts both ids appear in the table in reverse-chronological order
- Verifies via boto3 that two objects exist under the configured prefix and the first object's body starts with the RDB magic bytes (REDIS or VALKEY)

Notes for reviewers

The MicroCeph fixture is adapted from canonical/charmed-etcd-operator's tests/integration/backup/conftest.py
(3.6/edge). The single test deviation from etcd-operator: a --rdb magic-byte verification of the uploaded object replaces etcd's "restore on a new model" round-trip, since restore is not yet implemented.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Exercises the per-unit lock and any-unit guarantee by running backups from three different units and listing from a fourth. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

The committed fixtures could not pass: the TLS CA chain never reached the charm, MicroCeph RGW was never actually served over TLS (--ssl-certificate=auto is not valid and fell back to plaintext :80), the endpoint used loopback (unreachable from charm units), and credentials were set via a non-existent s3-integrator config option. - conftest.py: idempotent MicroCeph setup; generate a self-signed cert with SAN covering the host IP, serve RGW over TLS on 445 with it, and return a host-routable https endpoint plus that cert as the CA chain. - test_s3_backup.py: base64-encode the CA into the tls-ca-chain config; supply credentials through the sync-s3-credentials action; pin channel=9/edge; use the shared juju fixture; gate the action and the post-integrate wait on agents-idle to avoid "no actions defined" and pre-wired relation races. Verified end-to-end on microk8s: 1 passed in 174.77s. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

reneradoi

Only a few comments from my side, the main one is about always deploying the local version of the charm, not the one from Charmhub, because what's the point of testing then ;)

Thank you Mykola!

skourta

Tested against s3-integrator 2/edge. Everyting works. I left some comments.

skourta · 2026-06-30T10:52:08Z

+# Self-signed cert + key, persisted so repeated local runs reuse the exact
+# material the already-running gateway serves -- regenerating without
+# re-enabling RGW would break TLS verification against the old certificate.
+_CERT_DIR = Path.home() / ".cache" / "valkey-itest"


Why name it valkey-itest we can have it generic across products I usually run OS, etcd, and Valkey on the same microceph.

Renamed to ~/.cache/microceph-rgw — it's the MicroCeph gateway cert, so it's shared across OS/etcd/Valkey on one MicroCeph.

skourta · 2026-06-30T10:59:57Z

+
+def test_backup_and_list(juju: jubilant.Juju, microceph: dict, s3_bucket) -> None:
+    juju.deploy(APP_NAME, channel="9/edge", num_units=3, trust=True, base="ubuntu@24.04")
+    juju.deploy(S3_INTEGRATOR_APP, channel="latest/edge")


Let's please use 2/edge. I believe latest/edge is a very old one and not updated/maintained.

Switched to 2/edge.

skourta · 2026-06-30T11:07:50Z

+        lambda status: jubilant.all_agents_idle(status, S3_INTEGRATOR_APP),
+        timeout=600,
+    )
+    juju.run(


These are passed as a secret for version 2 of the s3-integrator

Done. Dropped sync-s3-credentials; now add-secret + grant-secret and set credentials= on s3-integrator 2/edge. tls-ca-chain config is unchanged (still present in v2).

- deploy the local charm under test (resources on k8s) instead of the Charmhub 9/edge release, so the suite actually exercises code changes - move s3-integrator to 2/edge and supply credentials through a Juju secret (credentials config), replacing the removed sync-s3-credentials action; tls-ca-chain config is unchanged - replace the bespoke _wait_active with the shared are_apps_active_and_agents_idle helper - fold the subprocess helpers onto subprocess.run(check=True) with try/except CalledProcessError so failures surface stderr - name the RGW cert cache dir microceph-rgw so a shared MicroCeph across products reuses one gateway certificate - resolve the three backup units dynamically instead of hardcoding Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

reneradoi

Looks good!

delgod marked this pull request as draft June 14, 2026 13:50

delgod mentioned this pull request Jun 24, 2026

DA262: S3 backup (create-backup + list-backups) #59

Merged

delgod and others added 2 commits June 27, 2026 12:25

test(integration): add MicroCeph + s3 bucket fixtures

fda324e

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

test(integration): end-to-end S3 backup + list against MicroCeph

4bd84ea

Exercises the per-unit lock and any-unit guarantee by running backups from three different units and listing from a fourth. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

delgod force-pushed the s3-backup-integration-tests branch from f8653d7 to 4bd84ea Compare June 27, 2026 10:25

delgod marked this pull request as ready for review June 29, 2026 13:20

reneradoi reviewed Jun 30, 2026

View reviewed changes

skourta reviewed Jun 30, 2026

View reviewed changes

reneradoi approved these changes Jul 6, 2026

View reviewed changes

Uh oh!

Conversation

delgod commented May 13, 2026

Summary

Depends on

What's in the PR

Notes for reviewers

Uh oh!

reneradoi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

skourta left a comment

Choose a reason for hiding this comment

Uh oh!

skourta Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

delgod Jul 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

skourta Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

delgod Jul 6, 2026

Choose a reason for hiding this comment

Uh oh!

skourta Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

delgod Jul 6, 2026

Choose a reason for hiding this comment

Uh oh!

reneradoi left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants