Add the GarageCluster and S3Connection resources

AGENTS.md

@@ -38,6 +38,9 @@ When in doubt about a pattern, look there first.
 - **Stack**: Java 25, Quarkus, the Quarkus Operator SDK and the fabric8 Kubernetes client.
 - **Generated artifacts**: the CRDs and the Helm chart are generated from the code at build time, into
   `operator/build/kubernetes` and `operator/build/helm`. Change the Java classes or `application.yml`, never the output.
+- **Backend**: Garage only, driven through its Admin API v2. `GarageService` in `core/adminapi` is what the
+  reconcilers call; `GarageAdminApi` is the HTTP client underneath. The exception is `GarageClusterReconciler`: it
+  builds the client from its own `adminEndpoint` and passes it to `GarageClusterService`.
 - **Tests**: `@QuarkusTest` integration tests against a K3s cluster and a real Garage node, both provided by Quarkus
   Dev Services, so Docker is required. Prefer a test against the real Garage over a mock: what matters is how the
   Admin API actually behaves.

README.md

@@ -12,6 +12,8 @@ It is the companion of the [AboutBits Garage Helm chart](https://github.com/abou
 | **Kubernetes** | 1.34+                                     |
 
 > **Note:** Kubernetes 1.34+ is required. The tests run against K3s 1.34, and older Kubernetes versions are end-of-life. The Helm chart refuses to install on an older cluster.
+>
+> **Note:** A [`GarageCluster`](docs/garage-cluster.md) needs Garage installed with the chart value `garage.singleNode: false`. With the default `--single-node`, Garage creates its layout itself and does not start again once the operator has applied a newer one. Without a `GarageCluster`, keep the default.
 
 ## Contribute
 
@@ -62,7 +64,7 @@ git config core.hooksPath .githooks
 
 ### Development
 
-You can run the operator in dev mode, against a throwaway K3s cluster, using:
+You can run the operator in dev mode, against a throwaway K3s cluster and a real Garage node, using:
 
 ```bash
 make run
@@ -82,6 +84,8 @@ make test
 ./gradlew :operator:test
 ```
 
+The tests are `@QuarkusTest` integration tests against real infrastructure provided by Quarkus Dev Services: a K3s cluster with the CRDs applied, and a Garage node started from `operator/src/main/docker/compose-devservices.yml`. The Garage image there is pinned to the version the Helm chart deploys, so the Admin API contract is exercised against the real thing.
+
 ## Information
 
 About Bits is a company based in South Tyrol, Italy. You can find more information about us on [our website](https://aboutbits.it).

docs/garage-cluster.md

@@ -0,0 +1,110 @@
+# GarageCluster
+
+A `GarageCluster` bootstraps and converges the **cluster layout** of a Garage installation.
+
+A Garage node started without `--single-node` knows about itself but has no role in any layout, and in that state it **serves no S3 traffic at all** — every request fails until a layout is assigned and applied. The `GarageCluster` assigns that layout.
+
+## Requirement: `garage.singleNode: false`
+
+By default the [AboutBits Garage Helm chart](https://github.com/aboutbits/helm-garage) starts Garage with `--single-node`. Garage then creates the layout itself, and refuses to start once the layout version is higher than 1. A `GarageCluster` applies a new layout version as soon as its spec differs from the role of the node, so with `--single-node` the next restart of the Garage Pod fails for good.
+
+Install the chart with `garage.singleNode: false` to manage the layout with a `GarageCluster`. Without a `GarageCluster`, keep the default: Garage creates its own layout, and [`S3Connection`](s3-connection.md), `Bucket` and `AccessKey` work the same. The chart's `bootstrap` option needs single-node mode, so with `garage.singleNode: false` create the bucket and the key with a `Bucket` and an `AccessKey` instead.
+
+The operator cannot see how Garage was started, so it cannot refuse a `GarageCluster` for a Garage in single-node mode.
+
+The operator drives Garage through its
+[Admin API v2](https://garagehq.deuxfleurs.fr/documentation/reference-manual/admin-api/), not through the S3 API — Garage implements neither IAM nor bucket policies, so buckets, keys and permissions are all Admin API concerns too.
+
+Once the layout is applied, an [`S3Connection`](s3-connection.md) pointed at the same cluster becomes
+`READY`, which is what unblocks buckets and access keys.
+
+## Scope
+
+This version manages **single-node clusters**, which is what the Helm chart deploys (`replication_factor = 1`, no clustering or node discovery). If the cluster reports more than one node, the operator refuses to touch the layout and reports `ERROR` rather than guessing a zone and capacity per node.
+
+Only one `GarageCluster` manages a Garage node. A second `GarageCluster` for the same node, in any namespace, reports `ERROR` and leaves the layout alone. Otherwise two resources with different layouts would apply their roles in turn, a new layout version at every reconcile. The node is recognized by its id, so two spellings of the same `adminEndpoint` count as the same Garage. When the first `GarageCluster` is deleted, the second one takes over at its next retry.
+
+Deleting a `GarageCluster` removes the operator's record of the cluster. It does **not** tear the storage cluster down: removing a node's role would trigger a rebalance and, on a single-node cluster, discard the data. There is therefore no reclaim policy on this resource.
+
+## Example
+
+```yaml
+apiVersion: garage.aboutbits.it/v1
+kind: GarageCluster
+metadata:
+  namespace: acme
+  name: acme
+spec:
+  adminEndpoint: http://acme-garage.acme.svc:3903
+  adminSecretRef:
+    name: acme-garage
+    key: admin_token
+  layout:
+    zone: default
+    capacity: 20Gi
+    tags:
+      - production
+```
+
+## Spec
+
+| Field            | Type                                                         | Description                                                                                                                                                                                                         | Required | Mutable |
+|------------------|--------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|---------|
+| `adminEndpoint`  | `string`                                                     | Base URL of the Garage Admin API, port 3903. The chart puts it on the same Service as the S3 API. Keep `service.type: ClusterIP`, because another type also makes the Admin API reachable from outside the cluster. | Yes      | Yes     |
+| `adminSecretRef` | [`SecretKeyRef`](#secretkeyref-adminsecretref)               | Reference to the `Opaque` Secret holding the Admin API bearer token. The chart creates one named after the release.                                                                                                 | Yes      | Yes     |
+| `layout`         | [`GarageClusterLayoutSpec`](#garageclusterlayoutspec-layout) | The role of the node in the cluster layout.                                                                                                                                                                         | Yes      | Yes     |
+
+### SecretKeyRef (`adminSecretRef`)
+
+| Field       | Type     | Description                                                                           | Required |
+|-------------|----------|---------------------------------------------------------------------------------------|----------|
+| `namespace` | `string` | Namespace of the Secret. If not specified, uses the namespace of the `GarageCluster`. | No       |
+| `name`      | `string` | Name of the Secret.                                                                   | Yes      |
+| `key`       | `string` | Key within the Secret. The chart stores the token under `admin_token`.                | Yes      |
+
+### GarageClusterLayoutSpec (`layout`)
+
+| Field      | Type            | Description                                                                                                                                                                                               | Required |
+|------------|-----------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|
+| `zone`     | `string`        | Zone the node is placed in. With `replication_factor = 1` this is effectively a label.                                                                                                                    | Yes      |
+| `capacity` | `string`        | Storage capacity to advertise, as a Kubernetes quantity such as `20Gi`. Should match the size of the chart's data volume. Garage uses it to weight partition distribution; it is not enforced as a limit. | Yes      |
+| `tags`     | `array[string]` | Tags to attach to the node. Informational; order is not significant.                                                                                                                                      | No       |
+
+A change to a `Mutable` field is applied at the next reconcile. A change to `layout` applies a new layout version. On a single node no data moves. A role that you change by hand with the Garage CLI is reset at the next reconcile, at most 10 minutes later.
+
+Note that the CRD schema carries no defaults — every required field has to be spelled out in the manifest.
+
+The token needs access to these Admin API endpoints: `GetClusterStatus`, `GetClusterHealth`, `GetClusterLayout`, `UpdateClusterLayout` and `ApplyClusterLayout`. The token of the Helm chart has access to all endpoints. A Garage admin token with a smaller scope must list the endpoints that it allows.
+
+## Status
+
+| Field                             | Description                                                                    |
+|-----------------------------------|--------------------------------------------------------------------------------|
+| `phase`                           | `PENDING`, `READY`, `ERROR`                                                    |
+| `layoutVersion`                   | Version number of the applied cluster layout. `0` means none has been applied. |
+| `nodeId`                          | Full-length identifier of the Garage node.                                     |
+| `health`                          | Cluster health as reported by Garage: `healthy`, `degraded` or `unavailable`.  |
+| `storageNodes` / `storageNodesUp` | Storage nodes in the layout, and how many are connected.                       |
+| `message`                         | Human-readable detail about the current state.                                 |
+
+`PENDING` is the normal state right after a `helm install`, while Garage is still starting and reports no nodes or a disconnected node; it is retried every 15 seconds.
+
+`ERROR` has three causes:
+
+- The Secret, the token or the endpoint is wrong, or Garage cannot be reached. The operator tries again every 60 seconds.
+- Another `GarageCluster` already manages the node. The operator tries again every 60 seconds, so this one takes over when the other one is deleted.
+- Garage reports more than one node. The operator does not try again before the next resync after 10 minutes, or a change of the spec.
+
+`health`, `storageNodes` and `storageNodesUp` are only set while the phase is `READY`, so another phase does not show an old value.
+
+## How reconciliation works
+
+1. Read `GetClusterStatus`. No nodes, or the node not connected, means Garage is still coming up →
+   `PENDING`, retry. A node that another `GarageCluster` already manages → `ERROR`.
+2. Compare the node's assigned role against the spec. Tags are compared as a set, and capacity is compared in bytes, so a reordered tag list or `20Gi` versus `21474836480` is not a change.
+3. If they match, stop — nothing is written to Garage.
+4. Otherwise read `GetClusterLayout` for the authoritative version number, stage the role with
+   `UpdateClusterLayout`, then commit it with `ApplyClusterLayout` at version + 1.
+5. Read `GetClusterHealth` to fill in the status, and report `READY`.
+
+Because step 2 short-circuits, reconciliation is idempotent: a `GarageCluster` whose layout already matches never creates a new layout version, no matter how often it is reconciled.

docs/s3-connection.md

@@ -0,0 +1,70 @@
+# S3Connection
+
+An `S3Connection` names a Garage Admin API and the token to administer it. Buckets, access keys and permissions reference a connection, which tells the operator where to create them.
+
+The operator's work is almost entirely **admin-plane**, and the admin plane is not part of the S3 API: Garage has no IAM and no bucket policies, and manages buckets, keys and their per-key, per-bucket grants through its Admin API only.
+
+## Relationship to GarageCluster
+
+For Garage the two are siblings, not alternatives:
+
+| Resource                             | Answers                                                     |
+|--------------------------------------|-------------------------------------------------------------|
+| [`GarageCluster`](garage-cluster.md) | Is the cluster itself set up? (layout assigned and applied) |
+| `S3Connection`                       | Can I administer object storage on this backend right now?  |
+
+A connection only reports `READY` when the backend can actually serve requests. A Garage node that answers its Admin API but has no layout reports health `unavailable`, and the connection stays
+`PENDING` with a message pointing at the `GarageCluster`. Because `Bucket` and key resources gate on their connection being `READY`, that one check keeps them from acting against a cluster that is not finished bootstrapping.
+
+Both resources reference the same admin endpoint and Secret. That duplication is deliberate — each is self-contained, and a backend the operator only *uses* does not need a `GarageCluster` at all.
+
+## Example
+
+```yaml
+apiVersion: garage.aboutbits.it/v1
+kind: S3Connection
+metadata:
+  namespace: acme
+  name: acme
+spec:
+  adminEndpoint: http://acme-garage.acme.svc:3903
+  adminSecretRef:
+    name: acme-garage
+    key: admin_token
+```
+
+## Spec
+
+| Field            | Type                                           | Description                                                                               | Required | Mutable |
+|------------------|------------------------------------------------|-------------------------------------------------------------------------------------------|----------|---------|
+| `adminEndpoint`  | `string`                                       | Base URL of the Garage **Admin** API, port 3903. This is not the S3 endpoint clients use. | Yes      | No      |
+| `adminSecretRef` | [`SecretKeyRef`](#secretkeyref-adminsecretref) | Reference to the `Opaque` Secret holding the Admin API bearer token.                      | Yes      | Yes     |
+
+### SecretKeyRef (`adminSecretRef`)
+
+| Field       | Type     | Description                                                                          | Required |
+|-------------|----------|--------------------------------------------------------------------------------------|----------|
+| `namespace` | `string` | Namespace of the Secret. If not specified, uses the namespace of the `S3Connection`. | No       |
+| `name`      | `string` | Name of the Secret.                                                                  | Yes      |
+| `key`       | `string` | Key within the Secret. The AboutBits Garage chart uses `admin_token`.                | Yes      |
+
+`adminEndpoint` is immutable, because buckets and access keys record the ids that they get from this Garage. A changed endpoint would create second ones on another Garage. Delete and recreate the `S3Connection` to point it at a different Garage. `adminSecretRef` can change, for example to rotate the token.
+
+The token needs access to the Admin API endpoint `GetClusterHealth`. The token of the Helm chart has access to all endpoints. A Garage admin token with a smaller scope must list the endpoints that it allows.
+
+## Status
+
+| Phase     | Meaning                                                                                                                                                 |
+|-----------|---------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `READY`   | The backend is reachable, the credentials are accepted, and it can serve requests.                                                                      |
+| `PENDING` | Reachable and authenticated, but not ready — for Garage, health is `unavailable`, usually because no layout has been applied. Retried every 30 seconds. |
+| `ERROR`   | Unreachable, or the credentials were rejected, or the Secret is missing. Retried every 60 seconds.                                                      |
+
+`message` carries the backend detail, for example
+`Garage cluster healthy [storageNodes=1, storageNodesUp=1, partitionsAllOk=256/256]`.
+
+Note that Garage's `degraded` status still counts as `READY`: a quorum of write nodes is available, so reads and writes succeed. Only `unavailable` blocks.
+
+## Health endpoint
+
+The operator's readiness endpoint does **not** probe backends. One unreachable Garage would otherwise make the operator itself unready and block its rollout, although the operator is fine and every other backend is still being served. The state of each backend is reported on its `S3Connection` instead.

operator/build.gradle.kts

@@ -51,6 +51,7 @@ dependencies {
      */
     testImplementation("io.quarkus:quarkus-junit")
     testImplementation("io.quarkus:quarkus-junit-mockito")
+    testImplementation("io.fabric8:kubernetes-server-mock")
     testImplementation("org.awaitility:awaitility")
     testImplementation(libs.assertj)
     testImplementation(libs.awsSdk.s3)
@@ -71,6 +72,11 @@ tasks.withType<Test>().configureEach {
     dependsOn(tasks.quarkusAppPartsBuild)
 
     jvmArgumentProviders.add(MockitoArgumentProvider(mockitoAgentProvider))
+
+    // The fabric8 client re-reads the kubeconfig on token refresh. A developer kubeconfig with an exec plugin
+    // (e.g. `aws eks get-token`) then runs that plugin before many requests, which slows every test against the
+    // Dev Services cluster. The tests use only the Dev Services cluster, so they must not see that file at all.
+    environment("KUBECONFIG", layout.buildDirectory.file("no-kubeconfig").get().asFile.absolutePath)
 }
 
 class MockitoArgumentProvider(

operator/src/main/docker/compose-devservices.yml

@@ -0,0 +1,19 @@
+services:
+  garage:
+    # Keep this in sync with the image the AboutBits Garage Helm chart deploys.
+    # https://github.com/aboutbits/helm-garage
+    image: dxflrs/garage:v2.4.1
+    # The chart's command with `garage.singleNode: false`, the mode a GarageCluster requires.
+    command: ["/garage", "server"]
+    volumes:
+      - ./garage.toml:/etc/garage.toml:ro
+    tmpfs:
+      - /var/lib/garage:rw,async,noatime
+    ports:
+      # S3 API
+      - "3900"
+      # Admin API
+      - "3903"
+    labels:
+      io.quarkus.devservices.compose.config_map.port.3900: garage.s3.port
+      io.quarkus.devservices.compose.config_map.port.3903: garage.admin.port