From e7831a3d471cd17d5c5975adab94f1ab1edad051 Mon Sep 17 00:00:00 2001 From: Andrei Kvapil Date: Thu, 8 Oct 2026 20:25:32 +0200 Subject: [PATCH 01/21] feat(site): add the blockstor.org landing page A single static page for blockstor.org, built from the README: what blockstor is, why it is built as a Kubernetes control plane, what works today and what does not, the release images and the documentation links. Plain HTML and CSS with no build step, light and dark from the system setting, and no analytics. Assisted-by: LLM Signed-off-by: Andrei Kvapil --- site/404.html | 29 ++ site/CNAME | 1 + site/img/blockstor-logo-black.svg | 6 + site/img/blockstor-logo-white.svg | 6 + site/img/favicon.svg | 6 + site/index.html | 214 ++++++++++++ site/robots.txt | 2 + site/style.css | 518 ++++++++++++++++++++++++++++++ 8 files changed, 782 insertions(+) create mode 100644 site/404.html create mode 100644 site/CNAME create mode 100644 site/img/blockstor-logo-black.svg create mode 100644 site/img/blockstor-logo-white.svg create mode 100644 site/img/favicon.svg create mode 100644 site/index.html create mode 100644 site/robots.txt create mode 100644 site/style.css diff --git a/site/404.html b/site/404.html new file mode 100644 index 00000000..0f1e9c28 --- /dev/null +++ b/site/404.html @@ -0,0 +1,29 @@ + + + + + +Page not found | Blockstor + + + + + + +
+
+ + + + +
+
+
+
+

This page does not exist.

+

The address may be mistyped, or the page has moved.

+

Go to the home page

+
+
+ + diff --git a/site/CNAME b/site/CNAME new file mode 100644 index 00000000..fb97e310 --- /dev/null +++ b/site/CNAME @@ -0,0 +1 @@ +blockstor.org diff --git a/site/img/blockstor-logo-black.svg b/site/img/blockstor-logo-black.svg new file mode 100644 index 00000000..746ef3fa --- /dev/null +++ b/site/img/blockstor-logo-black.svg @@ -0,0 +1,6 @@ + + + + + + diff --git a/site/img/blockstor-logo-white.svg b/site/img/blockstor-logo-white.svg new file mode 100644 index 00000000..1e4fb76a --- /dev/null +++ b/site/img/blockstor-logo-white.svg @@ -0,0 +1,6 @@ + + + + + + diff --git a/site/img/favicon.svg b/site/img/favicon.svg new file mode 100644 index 00000000..dd5e9375 --- /dev/null +++ b/site/img/favicon.svg @@ -0,0 +1,6 @@ + + + + + + diff --git a/site/index.html b/site/index.html new file mode 100644 index 00000000..a919dd32 --- /dev/null +++ b/site/index.html @@ -0,0 +1,214 @@ + + + + + +Blockstor: Kubernetes-native block storage with DRBD replication + + + + + + + + + + + +
+
+ + + + + +
+
+ +
+
+
+
+

A Kubernetes control plane for LVM and ZFS storage with DRBD replication.

+

Blockstor speaks a LINSTOR-compatible REST API, so the clients you already use keep working unchanged: linstor-csi, piraeus-operator, ha-controller, golinstor and the linstor CLI.

+

The difference is underneath. Desired state lives in CRDs, and a set of controller-runtime reconcilers drive the cluster toward it. There is no external database to back up, no in-memory state to lose on restart, and no controller-to-node polling to fall behind.

+
+ Get started + View on GitHub +
+
+ +
+ + One resource definition in the Kubernetes API, reconciled into three DRBD replicas on three nodes + + + ResourceDefinition + + + + + + + + + node-1 + + + + satellite + + + + node-2 + + + + satellite + + + + node-3 + + + + satellite + + + + + + +
Each satellite watches its own slice of the CRDs and writes what it observes back through Status, while DRBD keeps the replicas in sync.
+
+
+
+ +
+
+

Why blockstor

+
+
+

Kubernetes-native architecture

+
+
Reconciliation control plane
+
The state of truth lives in Kubernetes CRDs. The controller and the satellites are controller-runtime managers with watch-based informers, declarative reconcile loops and Status written by Server-Side Apply. Desired and observed state converge on their own.
+
First-class CRDs
+
Resource, ResourceDefinition, ResourceGroup, StoragePool, Snapshot, Node and PhysicalDevice are meant to be read, and where it makes sense written, by other operators: tenant operators, GitOps tooling, monitoring and admission webhooks.
+
Per-node satellite as a controller
+
Each satellite watches the CRDs for its own node and reports what it observes through Status. There is no gRPC dispatch from a central controller.
+
+
+
+

Ecosystem fit

+
+
Built in Go
+
Go is what the apiserver, the kubelet, etcd, most CSI drivers and controller-runtime are written in, so blockstor shares their tooling, libraries and contributors.
+
+
+
+

Small footprint

+
+
No runtime to ship
+
The controller, the apiserver and the satellite are statically linked Go binaries, with no JVM, and DRBD is a Linux kernel module rather than a userspace daemon. The images are multi-arch, linux/amd64 and linux/arm64, so blockstor fits on edge and small ARM nodes.
+
+
+
+
+
+ +
+
+

Status

+
+
+

Works today

+
    +
  • Replicated volumes over DRBD, on LVM, LVM-thin, ZFS, ZFS-thin or file backends
    • +
  • Running without DRBD: plain local storage, single-replica diskful or diskless
    • +
  • LUKS encryption: volume-level encryption at rest
    • +
  • Autoplacement with constraints: zones, node properties, replicas on different nodes
    • +
  • TieBreaker and quorum policies
    • +
  • Snapshots: create, restore as a new resource, roll back, and clone
    • +
  • Intra-cluster snapshot shipping (zfs send/recv, thin-send-recv) for clone and add-replica
    • +
  • Online volume resize
    • +
  • Device-pool creation from physical disks (physical-storage create-device-pool)
    • +
  • A LINSTOR-compatible REST API for the whole client ecosystem, served over mTLS
    • +
+
+
+

Not implemented

+

The API answers these with 501 Not Implemented.

+
    +
  • Cross-cluster snapshot shipping (disaster recovery)
    • +
  • Backup create, restore, ship and abort, and the backup queue
    • +
  • Schedules (cron-driven backups)
    • +
  • Remote backends: S3, LINSTOR remotes
    • +
  • Extra storage providers: SPDK, NVMe-oF, OpenFlex, Exos
    • +
+
+
+

On the roadmap

+
    +
  • Bring-your-own-key encryption: operator-managed Secret references in the spec, instead of a controller-owned passphrase bag
    • +
  • A migration tool from LINSTOR that adopts an existing LINSTOR cluster's resources into blockstor in place
    • +
  • Shared-LUN provisioning: thick LVM plus thin qcow2-on-LVM, no filesystem layer
    • +
  • VDUSE backend via qemu-storage-daemon, for shared-SAN Kubernetes
    • +
+
+
+
+
+ +
+
+
+

Get started

+

Blockstor installs onto an existing cluster and is driven with the standard linstor client and piraeus linstor-csi. The usage guide walks through installing the control plane, registering nodes and pools, and wiring up CSI.

+

Read the usage guide

+
+
+

Images

+

The three images are published to GHCR on every release.

+
    +
  • ghcr.io/cozystack/blockstor-controllerthe reconcilers
    • +
  • ghcr.io/cozystack/blockstor-apiserverthe LINSTOR-compatible REST API, over mTLS
    • +
  • ghcr.io/cozystack/blockstor-satellitethe per-node DRBD and storage agent
    • +
+
+
+
+ +
+
+

Documentation

+
    +
  • UsageInstall and operate blockstor with the linstor client and piraeus linstor-csi.
    • +
  • ArchitectureThe load-bearing design decisions.
    • +
  • Layer stackDRBD, LUKS and STORAGE compositions.
    • +
  • ContributingRepository layout and the local Talos and QEMU dev stand.
    • +
+
+
+ +
+
+

Acknowledgements

+

Blockstor was inspired by LINBIT's LINSTOR, and it operates DRBD to provide block-level replication. It deliberately speaks LINSTOR's API so that the ecosystem the LINSTOR community has built keeps working unchanged. Heartfelt thanks to LINBIT and to the wider DRBD, LINSTOR and Piraeus community.

+
+
+
+ +
+
+

Blockstor is licensed under Apache 2.0. The code is provided as-is with no warranties. Third-party Go modules and their licenses are listed in THIRD_PARTY_LICENSES.md.

+

LINSTOR, LINBIT and DRBD are trademarks or registered trademarks of LINBIT. Blockstor is an independent project and is not affiliated with, endorsed by, or sponsored by LINBIT.

+
+
+ + diff --git a/site/robots.txt b/site/robots.txt new file mode 100644 index 00000000..c2a49f4f --- /dev/null +++ b/site/robots.txt @@ -0,0 +1,2 @@ +User-agent: * +Allow: / diff --git a/site/style.css b/site/style.css new file mode 100644 index 00000000..ab52cd3a --- /dev/null +++ b/site/style.css @@ -0,0 +1,518 @@ +:root { + --paper: #f3f5f9; + --band: #e7ebf2; + --ink: #15213b; + --muted: #4a5672; + --rule: #cfd6e3; + --accent: #2f46d6; + --accent-ink: #ffffff; + --code-bg: #e2e7f0; + --block: #2f46d6; + --block-soft: #c9d1f5; + --sans: "IBM Plex Sans", system-ui, -apple-system, "Segoe UI", sans-serif; + --mono: "IBM Plex Mono", ui-monospace, SFMono-Regular, Menlo, monospace; +} + +@media (prefers-color-scheme: dark) { + :root { + --paper: #0e1524; + --band: #141d30; + --ink: #e8edf6; + --muted: #a3aec4; + --rule: #2a3651; + --accent: #93a4ff; + --accent-ink: #0e1524; + --code-bg: #1d2840; + --block: #93a4ff; + --block-soft: #2b3a66; + } +} + +*, +*::before, +*::after { + box-sizing: border-box; +} + +html { + scroll-behavior: smooth; +} + +@media (prefers-reduced-motion: reduce) { + html { + scroll-behavior: auto; + } +} + +body { + margin: 0; + background: var(--paper); + color: var(--ink); + font-family: var(--sans); + font-size: 1.0625rem; + line-height: 1.6; + -webkit-text-size-adjust: 100%; +} + +img, +svg { + max-width: 100%; + height: auto; +} + +a { + color: var(--accent); + text-underline-offset: 0.18em; +} + +a:hover { + text-decoration-thickness: 2px; +} + +:focus-visible { + outline: 3px solid var(--accent); + outline-offset: 3px; + border-radius: 2px; +} + +code { + font-family: var(--mono); + font-size: 0.88em; + background: var(--code-bg); + padding: 0.08em 0.32em; + border-radius: 3px; + overflow-wrap: anywhere; +} + +h1, +h2, +h3 { + line-height: 1.2; + margin: 0 0 0.6em; +} + +h1 { + font-size: clamp(1.9rem, 4.2vw, 3rem); + font-weight: 600; + letter-spacing: -0.015em; + max-width: 22ch; +} + +h2 { + font-size: clamp(1.5rem, 2.6vw, 2rem); + font-weight: 600; + letter-spacing: -0.01em; +} + +h3 { + font-size: 1.15rem; + font-weight: 600; +} + +p { + margin: 0 0 1em; + max-width: 68ch; +} + +.wrap { + width: 100%; + max-width: 72rem; + margin: 0 auto; + padding: 0 1rem; +} + +@media (min-width: 40rem) { + .wrap { + padding: 0 2rem; + } +} + +.narrow p { + max-width: 68ch; +} + +.skip { + position: absolute; + left: -999px; + top: 0.5rem; + background: var(--accent); + color: var(--accent-ink); + padding: 0.5rem 0.9rem; + z-index: 10; +} + +.skip:focus { + left: 0.5rem; +} + +/* Header */ + +.top { + border-bottom: 1px solid var(--rule); +} + +.top-row { + display: flex; + flex-wrap: wrap; + align-items: center; + justify-content: space-between; + gap: 0.75rem 1.5rem; + padding-top: 1rem; + padding-bottom: 1rem; +} + +.brand { + display: inline-flex; +} + +.logo { + height: 28px; + width: auto; +} + +.logo-dark { + display: none; +} + +@media (prefers-color-scheme: dark) { + .logo-light { + display: none; + } + + .logo-dark { + display: block; + } +} + +nav { + display: flex; + flex-wrap: wrap; + gap: 0.25rem 1.25rem; +} + +nav a { + color: var(--ink); + text-decoration: none; + font-weight: 500; +} + +nav a:hover { + color: var(--accent); +} + +/* Hero */ + +.hero { + padding: 3.5rem 0 4rem; +} + +.hero-grid { + display: grid; + gap: 2.5rem; + align-items: center; +} + +@media (min-width: 56rem) { + .hero-grid { + grid-template-columns: 1.15fr 1fr; + gap: 3.5rem; + } +} + +.lede { + font-size: 1.2rem; + color: var(--ink); +} + +.hero-copy p:not(.lede) { + color: var(--muted); +} + +.actions { + display: flex; + flex-wrap: wrap; + gap: 0.75rem; + margin-top: 1.75rem; +} + +.btn { + display: inline-block; + padding: 0.65rem 1.15rem; + border: 1.5px solid var(--accent); + border-radius: 4px; + color: var(--accent); + font-weight: 600; + text-decoration: none; +} + +.btn:hover { + background: var(--band); +} + +.btn-primary { + background: var(--accent); + color: var(--accent-ink); +} + +.btn-primary:hover { + background: var(--accent); + filter: brightness(1.1); +} + +/* The replication figure: the one picture on the page. */ + +.replicas { + margin: 0; +} + +.replicas svg { + display: block; + width: 100%; + max-width: 30rem; + margin: 0 auto; + font-family: var(--mono); +} + +.replicas text { + fill: var(--ink); + font-size: 13px; +} + +.replicas .sub { + fill: var(--muted); + font-size: 12px; +} + +.replicas .crd rect { + fill: var(--paper); + stroke: var(--ink); + stroke-width: 1.5; +} + +.replicas .node > rect:first-child { + fill: var(--band); + stroke: var(--rule); + stroke-width: 1.5; +} + +.replicas .blk { + fill: var(--block); +} + +.replicas .links path { + fill: none; + stroke: var(--muted); + stroke-width: 1.5; +} + +.replicas .sync path { + fill: none; + stroke: var(--block); + stroke-width: 2; + stroke-dasharray: 4 4; +} + +@media (prefers-reduced-motion: no-preference) { + .replicas .sync path { + animation: sync 1.6s linear infinite; + } +} + +@keyframes sync { + to { + stroke-dashoffset: -16; + } +} + +.replicas figcaption { + margin: 1rem auto 0; + max-width: 30rem; + color: var(--muted); + font-size: 0.95rem; +} + +/* Sections */ + +main > section { + padding: 3.5rem 0; +} + +.band { + background: var(--band); +} + +.why-grid { + display: grid; + gap: 2rem; +} + +@media (min-width: 56rem) { + .why-grid { + grid-template-columns: 1.6fr 1fr 1fr; + gap: 2.5rem; + } +} + +dl { + margin: 0; +} + +dt { + font-weight: 600; + margin-top: 1rem; +} + +dt:first-child { + margin-top: 0; +} + +dd { + margin: 0.25rem 0 0; + color: var(--muted); +} + +/* Status: the marker shape says the state, so it reads without colour. */ + +.status-grid { + display: grid; + gap: 2rem; +} + +@media (min-width: 56rem) { + .status-grid { + grid-template-columns: 1.4fr 1fr 1fr; + gap: 2.5rem; + } +} + +.status ul { + list-style: none; + margin: 0; + padding: 0; +} + +.status li { + position: relative; + padding-left: 1.6rem; + margin-bottom: 0.6rem; +} + +.status li::before { + content: ""; + position: absolute; + left: 0; + top: 0.42em; + width: 0.8rem; + height: 0.8rem; + border-radius: 2px; +} + +.works li::before { + background: var(--block); +} + +.missing li::before { + border: 1.5px solid var(--muted); +} + +.planned li::before { + border: 1.5px dashed var(--block); +} + +.note { + color: var(--muted); + font-size: 0.95rem; +} + +/* Get started */ + +.start-grid { + display: grid; + gap: 2rem; +} + +@media (min-width: 56rem) { + .start-grid { + grid-template-columns: 1fr 1fr; + gap: 3rem; + } +} + +.images { + list-style: none; + margin: 0; + padding: 0; +} + +.images li { + display: flex; + flex-direction: column; + gap: 0.15rem; + padding: 0.75rem 0; + border-top: 1px solid var(--rule); +} + +.images li:last-child { + border-bottom: 1px solid var(--rule); +} + +.images code { + background: none; + padding: 0; + font-size: 0.92rem; +} + +.images span { + color: var(--muted); + font-size: 0.95rem; +} + +/* Documentation */ + +.docs { + list-style: none; + margin: 0; + padding: 0; + display: grid; + gap: 1.25rem 2.5rem; +} + +@media (min-width: 40rem) { + .docs { + grid-template-columns: 1fr 1fr; + } +} + +.docs li { + display: flex; + flex-direction: column; + gap: 0.2rem; +} + +.docs a { + font-weight: 600; + font-size: 1.1rem; +} + +.docs span { + color: var(--muted); +} + +/* Footer */ + +.foot { + border-top: 1px solid var(--rule); + padding: 2rem 0 2.5rem; + color: var(--muted); + font-size: 0.92rem; +} + +.foot p { + max-width: 80ch; +} + +/* 404 */ + +.lost { + padding: 5rem 0; +} From 581bc5d90c638f5977ff010c2b73e9568c018f08 Mon Sep 17 00:00:00 2001 From: Andrei Kvapil Date: Thu, 8 Oct 2026 20:26:02 +0200 Subject: [PATCH 02/21] ci(site): build deploy previews of the site on Netlify Pull requests get a Netlify preview of site/, the way cozystack.io reviews its site, while production stays on GitHub Pages. The preview is skipped when a pull request changes nothing under site/. Assisted-by: LLM Signed-off-by: Andrei Kvapil --- netlify.toml | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 netlify.toml diff --git a/netlify.toml b/netlify.toml new file mode 100644 index 00000000..dba41573 --- /dev/null +++ b/netlify.toml @@ -0,0 +1,21 @@ +# Netlify builds deploy previews of the blockstor.org site for pull +# requests. Production is GitHub Pages (.github/workflows/pages.yml). +# +# The site is plain static files under site/, so there is no build command: +# Netlify publishes the directory as it is. + +[build] + base = "site" + publish = "." + # Skip the preview when nothing under site/ or this file changed. The + # command runs from the base directory and exit 0 skips the build, so a + # missing or identical cached ref forces one. + ignore = ''' + if [ -z "$CACHED_COMMIT_REF" ] || [ "$CACHED_COMMIT_REF" = "$COMMIT_REF" ]; then exit 1; fi + git diff --quiet "$CACHED_COMMIT_REF" "$COMMIT_REF" -- . ../netlify.toml + ''' + +[[redirects]] + from = "/*" + to = "/404.html" + status = 404 From c570d263f4767db2c015b5e990c0dc1d594e9965 Mon Sep 17 00:00:00 2001 From: Andrei Kvapil Date: Thu, 8 Oct 2026 20:26:28 +0200 Subject: [PATCH 03/21] ci(site): publish the site to GitHub Pages from main blockstor.org is served from GitHub Pages, as cozystack.io is. A push to main that touches site/ publishes the directory as it is; there is nothing to build. Assisted-by: LLM Signed-off-by: Andrei Kvapil --- .github/workflows/pages.yml | 50 +++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100644 .github/workflows/pages.yml diff --git a/.github/workflows/pages.yml b/.github/workflows/pages.yml new file mode 100644 index 00000000..c06a2ac3 --- /dev/null +++ b/.github/workflows/pages.yml @@ -0,0 +1,50 @@ +name: Deploy site to Pages + +# blockstor.org is the static site under site/. It is published as is, with +# no build step; Netlify builds the deploy previews (netlify.toml). +on: + push: + branches: [main] + paths: + - 'site/**' + - '.github/workflows/pages.yml' + workflow_dispatch: + +permissions: + contents: read + pages: write + id-token: write + +# One deployment at a time. A queued run is skipped for the newest one, but +# a deployment in progress is allowed to finish. +concurrency: + group: pages + cancel-in-progress: false + +jobs: + build: + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + with: + persist-credentials: false + + - name: Setup Pages + uses: actions/configure-pages@45bfe0192ca1faeb007ade9deae92b16b8254a0d # v6.0.0 + + - name: Upload artifact + uses: actions/upload-pages-artifact@fc324d3547104276b827a68afc52ff2a11cc49c9 # v5.0.0 + with: + path: ./site + + deploy: + environment: + name: github-pages + url: ${{ steps.deployment.outputs.page_url }} + runs-on: ubuntu-latest + needs: build + steps: + - name: Deploy to GitHub Pages + id: deployment + uses: actions/deploy-pages@368f82528645a54fb793d4d04e342629a3f51346 # v5.0.1 From 2ede5043c692503d80fe6d9bc3e4760c01454155 Mon Sep 17 00:00:00 2001 From: Andrei Kvapil Date: Thu, 8 Oct 2026 20:26:53 +0200 Subject: [PATCH 04/21] ci: skip the test workflows for changes to the site alone A pull request or a push that only touches site/ or netlify.toml has nothing for the unit, integration, contract, lint or e2e lanes to test, and the e2e lanes are the expensive ones. The license check and the release workflow are left as they are. Assisted-by: LLM Signed-off-by: Andrei Kvapil --- .github/workflows/contract.yml | 4 ++++ .github/workflows/integration.yml | 4 ++++ .github/workflows/lint.yml | 4 ++++ .github/workflows/pull-request.yml | 4 ++++ .github/workflows/test-e2e.yml | 4 ++++ .github/workflows/test.yml | 4 ++++ 6 files changed, 24 insertions(+) diff --git a/.github/workflows/contract.yml b/.github/workflows/contract.yml index 670f4c60..ae29fd63 100644 --- a/.github/workflows/contract.yml +++ b/.github/workflows/contract.yml @@ -17,6 +17,10 @@ on: # merges still exercise contract tests. push: branches: [main] + # The site under site/ is published on its own and has no tests. + paths-ignore: + - 'site/**' + - 'netlify.toml' permissions: {} diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.yml index 197442fd..3d68099e 100644 --- a/.github/workflows/integration.yml +++ b/.github/workflows/integration.yml @@ -16,6 +16,10 @@ on: # merges still exercise the envtest harness. push: branches: [main] + # The site under site/ is published on its own and has no tests. + paths-ignore: + - 'site/**' + - 'netlify.toml' permissions: {} diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index 69a8a32f..3a8fd1cc 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -7,6 +7,10 @@ name: Lint on: push: branches: [main] + # The site under site/ is published on its own and has no tests. + paths-ignore: + - 'site/**' + - 'netlify.toml' permissions: {} diff --git a/.github/workflows/pull-request.yml b/.github/workflows/pull-request.yml index d97aedb8..8417d2ac 100644 --- a/.github/workflows/pull-request.yml +++ b/.github/workflows/pull-request.yml @@ -26,6 +26,10 @@ name: Pull Request on: pull_request: types: [opened, synchronize, reopened] + # The site under site/ is published on its own and has no tests. + paths-ignore: + - 'site/**' + - 'netlify.toml' # Cancel in-flight runs for the same PR when a new push arrives — saves # runner minutes on rapid force-push iterations. diff --git a/.github/workflows/test-e2e.yml b/.github/workflows/test-e2e.yml index ed62a118..82493ae4 100644 --- a/.github/workflows/test-e2e.yml +++ b/.github/workflows/test-e2e.yml @@ -7,6 +7,10 @@ name: E2E Tests on: push: branches: [main] + # The site under site/ is published on its own and has no tests. + paths-ignore: + - 'site/**' + - 'netlify.toml' permissions: {} diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index ab968f7a..5084e743 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -7,6 +7,10 @@ name: Tests on: push: branches: [main] + # The site under site/ is published on its own and has no tests. + paths-ignore: + - 'site/**' + - 'netlify.toml' permissions: {} From ffadc5ab2b0571b3226cebfefed63c2a5cf6b565 Mon Sep 17 00:00:00 2001 From: Andrei Kvapil Date: Thu, 8 Oct 2026 23:35:12 +0200 Subject: [PATCH 05/21] feat(site): move the site onto Hugo and Docsy The project documentation is going to live on blockstor.org, which needs navigation, search and per-page editing that a hand-written page does not give. The site now builds with Hugo and Docsy, the stack cozystack.io runs, so both sites are maintained the same way. Docsy is pulled as a Hugo module with its own go.mod under site/, which keeps it out of the repository's Go module. Assisted-by: LLM Signed-off-by: Andrei Kvapil --- .gitignore | 6 + site/404.html | 29 - site/CNAME | 1 - site/README.md | 11 + .../icons/logo.svg} | 0 site/assets/scss/_styles_project.scss | 9 + site/assets/scss/_variables_project.scss | 3 + site/go.mod | 8 + site/go.sum | 8 + site/hugo.yaml | 70 ++ site/index.html | 214 ---- site/layouts/_partials/favicons.html | 1 + site/package-lock.json | 912 ++++++++++++++++++ site/package.json | 8 + site/{img => static}/favicon.svg | 0 .../{ => static}/img/blockstor-logo-black.svg | 0 site/static/img/blockstor-logo-white.svg | 6 + site/{ => static}/robots.txt | 0 site/style.css | 518 ---------- 19 files changed, 1042 insertions(+), 762 deletions(-) delete mode 100644 site/404.html delete mode 100644 site/CNAME create mode 100644 site/README.md rename site/{img/blockstor-logo-white.svg => assets/icons/logo.svg} (100%) create mode 100644 site/assets/scss/_styles_project.scss create mode 100644 site/assets/scss/_variables_project.scss create mode 100644 site/go.mod create mode 100644 site/go.sum create mode 100644 site/hugo.yaml delete mode 100644 site/index.html create mode 100644 site/layouts/_partials/favicons.html create mode 100644 site/package-lock.json create mode 100644 site/package.json rename site/{img => static}/favicon.svg (100%) rename site/{ => static}/img/blockstor-logo-black.svg (100%) create mode 100644 site/static/img/blockstor-logo-white.svg rename site/{ => static}/robots.txt (100%) delete mode 100644 site/style.css diff --git a/.gitignore b/.gitignore index bebae282..32e8dace 100644 --- a/.gitignore +++ b/.gitignore @@ -25,3 +25,9 @@ .claude/ pkg/satellite/pvc-dispatch-adjust.res linstor-migrate + +# blockstor.org site build output +site/public/ +site/resources/ +site/node_modules/ +site/.hugo_build.lock diff --git a/site/404.html b/site/404.html deleted file mode 100644 index 0f1e9c28..00000000 --- a/site/404.html +++ /dev/null @@ -1,29 +0,0 @@ - - - - - -Page not found | Blockstor - - - - - - -
-
- - - - -
-
-
-
-

This page does not exist.

-

The address may be mistyped, or the page has moved.

-

Go to the home page

-
-
- - diff --git a/site/CNAME b/site/CNAME deleted file mode 100644 index fb97e310..00000000 --- a/site/CNAME +++ /dev/null @@ -1 +0,0 @@ -blockstor.org diff --git a/site/README.md b/site/README.md new file mode 100644 index 00000000..0324fc1a --- /dev/null +++ b/site/README.md @@ -0,0 +1,11 @@ +# blockstor.org + +The project website and documentation, built with [Hugo](https://gohugo.io/) and the [Docsy](https://www.docsy.dev/) theme. Production is published to GitHub Pages from `main`; pull requests get a Netlify deploy preview. + +## Run it locally + +You need Hugo extended (0.161 or newer), Go and Node.js 22. From this directory run `npm install` once, then `hugo server` and open http://localhost:1313/. + +## Where things go + +Documentation pages live under `content/en/docs/`, one markdown file per page with `title`, `weight` and `description` in the front matter; the weight sets the order in the sidebar. The landing page is `content/en/_index.md`. Write one continuous line per paragraph. diff --git a/site/img/blockstor-logo-white.svg b/site/assets/icons/logo.svg similarity index 100% rename from site/img/blockstor-logo-white.svg rename to site/assets/icons/logo.svg diff --git a/site/assets/scss/_styles_project.scss b/site/assets/scss/_styles_project.scss new file mode 100644 index 00000000..edd2e46a --- /dev/null +++ b/site/assets/scss/_styles_project.scss @@ -0,0 +1,9 @@ +// The logo is the full wordmark, so the site name next to it is dropped. +.navbar-brand__name { + display: none; +} + +.navbar-brand__logo svg { + height: 1.75rem; + width: auto; +} diff --git a/site/assets/scss/_variables_project.scss b/site/assets/scss/_variables_project.scss new file mode 100644 index 00000000..beb25a96 --- /dev/null +++ b/site/assets/scss/_variables_project.scss @@ -0,0 +1,3 @@ +// Brand colour taken from the logo. +$primary: #1b2a4a; +$secondary: #3b5b8c; diff --git a/site/go.mod b/site/go.mod new file mode 100644 index 00000000..f52a47a0 --- /dev/null +++ b/site/go.mod @@ -0,0 +1,8 @@ +module github.com/cozystack/blockstor/site + +go 1.23 + +require ( + github.com/google/docsy v0.12.0 // indirect + github.com/google/docsy/dependencies v0.7.2 // indirect +) diff --git a/site/go.sum b/site/go.sum new file mode 100644 index 00000000..a130a73d --- /dev/null +++ b/site/go.sum @@ -0,0 +1,8 @@ +github.com/FortAwesome/Font-Awesome v0.0.0-20230327165841-0698449d50f2/go.mod h1:IUgezN/MFpCDIlFezw3L8j83oeiIuYoj28Miwr/KUYo= +github.com/FortAwesome/Font-Awesome v0.0.0-20241216213156-af620534bfc3/go.mod h1:IUgezN/MFpCDIlFezw3L8j83oeiIuYoj28Miwr/KUYo= +github.com/google/docsy v0.12.0 h1:CddZKL39YyJzawr8GTVaakvcUTCJRAAYdz7W0qfZ2P4= +github.com/google/docsy v0.12.0/go.mod h1:1bioDqA493neyFesaTvQ9reV0V2vYy+xUAnlnz7+miM= +github.com/google/docsy/dependencies v0.7.2 h1:+t5ufoADQAj4XneFphz4A+UU0ICAxmNaRHVWtMYXPSI= +github.com/google/docsy/dependencies v0.7.2/go.mod h1:gihhs5gmgeO+wuoay4FwOzob+jYJVyQbNaQOh788lD4= +github.com/twbs/bootstrap v5.2.3+incompatible/go.mod h1:fZTSrkpSf0/HkL0IIJzvVspTt1r9zuf7XlZau8kpcY0= +github.com/twbs/bootstrap v5.3.6+incompatible/go.mod h1:fZTSrkpSf0/HkL0IIJzvVspTt1r9zuf7XlZau8kpcY0= diff --git a/site/hugo.yaml b/site/hugo.yaml new file mode 100644 index 00000000..f55062f8 --- /dev/null +++ b/site/hugo.yaml @@ -0,0 +1,70 @@ +baseURL: https://blockstor.org/ +title: Blockstor +enableGitInfo: true + +# Since Hugo v0.161.0 the Node asset tools (PostCSS here) run under Node's +# permission model, with read access limited to the project directory. Docsy +# arrives as a Hugo module and its SCSS is served from the module cache +# outside the project tree, so read access is widened, as on cozystack.io. +security: + node: + permissions: + allowRead: + - '*' + +module: + proxy: direct + hugoVersion: + extended: true + imports: + - path: github.com/google/docsy + - path: github.com/google/docsy/dependencies + mounts: + - source: content/en + target: content + - source: assets + target: assets + - source: layouts + target: layouts + - source: static + target: static + +markup: + goldmark: + renderer: + unsafe: true + highlight: + style: friendly + +menu: + main: + - name: GitHub + weight: 50 + url: https://github.com/cozystack/blockstor + pre: + +params: + description: A Kubernetes control plane for LVM and ZFS storage with DRBD replication. + copyright: + authors: The Blockstor Authors + from_year: 2026 + github_repo: https://github.com/cozystack/blockstor + github_subdir: site + github_branch: main + offlineSearch: true + prism_syntax_highlighting: false + ui: + navbar_logo: true + navbar_translucent_over_cover_disable: false + sidebar_menu_compact: false + sidebar_search_disable: false + breadcrumb_disable: false + showLightDarkModeMenu: true + feedback: + enable: false + links: + developer: + - name: GitHub + url: https://github.com/cozystack/blockstor + icon: fa-brands fa-github + desc: Code, issues and releases diff --git a/site/index.html b/site/index.html deleted file mode 100644 index a919dd32..00000000 --- a/site/index.html +++ /dev/null @@ -1,214 +0,0 @@ - - - - - -Blockstor: Kubernetes-native block storage with DRBD replication - - - - - - - - - - - -
-
- - - - - -
-
- -
-
-
-
-

A Kubernetes control plane for LVM and ZFS storage with DRBD replication.

-

Blockstor speaks a LINSTOR-compatible REST API, so the clients you already use keep working unchanged: linstor-csi, piraeus-operator, ha-controller, golinstor and the linstor CLI.

-

The difference is underneath. Desired state lives in CRDs, and a set of controller-runtime reconcilers drive the cluster toward it. There is no external database to back up, no in-memory state to lose on restart, and no controller-to-node polling to fall behind.

-
- Get started - View on GitHub -
-
- -
- - One resource definition in the Kubernetes API, reconciled into three DRBD replicas on three nodes - - - ResourceDefinition - - - - - - - - - node-1 - - - - satellite - - - - node-2 - - - - satellite - - - - node-3 - - - - satellite - - - - - - -
Each satellite watches its own slice of the CRDs and writes what it observes back through Status, while DRBD keeps the replicas in sync.
-
-
-
- -
-
-

Why blockstor

-
-
-

Kubernetes-native architecture

-
-
Reconciliation control plane
-
The state of truth lives in Kubernetes CRDs. The controller and the satellites are controller-runtime managers with watch-based informers, declarative reconcile loops and Status written by Server-Side Apply. Desired and observed state converge on their own.
-
First-class CRDs
-
Resource, ResourceDefinition, ResourceGroup, StoragePool, Snapshot, Node and PhysicalDevice are meant to be read, and where it makes sense written, by other operators: tenant operators, GitOps tooling, monitoring and admission webhooks.
-
Per-node satellite as a controller
-
Each satellite watches the CRDs for its own node and reports what it observes through Status. There is no gRPC dispatch from a central controller.
-
-
-
-

Ecosystem fit

-
-
Built in Go
-
Go is what the apiserver, the kubelet, etcd, most CSI drivers and controller-runtime are written in, so blockstor shares their tooling, libraries and contributors.
-
-
-
-

Small footprint

-
-
No runtime to ship
-
The controller, the apiserver and the satellite are statically linked Go binaries, with no JVM, and DRBD is a Linux kernel module rather than a userspace daemon. The images are multi-arch, linux/amd64 and linux/arm64, so blockstor fits on edge and small ARM nodes.
-
-
-
-
-
- -
-
-

Status

-
-
-

Works today

-
    -
  • Replicated volumes over DRBD, on LVM, LVM-thin, ZFS, ZFS-thin or file backends
    • -
  • Running without DRBD: plain local storage, single-replica diskful or diskless
    • -
  • LUKS encryption: volume-level encryption at rest
    • -
  • Autoplacement with constraints: zones, node properties, replicas on different nodes
    • -
  • TieBreaker and quorum policies
    • -
  • Snapshots: create, restore as a new resource, roll back, and clone
    • -
  • Intra-cluster snapshot shipping (zfs send/recv, thin-send-recv) for clone and add-replica
    • -
  • Online volume resize
    • -
  • Device-pool creation from physical disks (physical-storage create-device-pool)
    • -
  • A LINSTOR-compatible REST API for the whole client ecosystem, served over mTLS
    • -
-
-
-

Not implemented

-

The API answers these with 501 Not Implemented.

-
    -
  • Cross-cluster snapshot shipping (disaster recovery)
    • -
  • Backup create, restore, ship and abort, and the backup queue
    • -
  • Schedules (cron-driven backups)
    • -
  • Remote backends: S3, LINSTOR remotes
    • -
  • Extra storage providers: SPDK, NVMe-oF, OpenFlex, Exos
    • -
-
-
-

On the roadmap

-
    -
  • Bring-your-own-key encryption: operator-managed Secret references in the spec, instead of a controller-owned passphrase bag
    • -
  • A migration tool from LINSTOR that adopts an existing LINSTOR cluster's resources into blockstor in place
    • -
  • Shared-LUN provisioning: thick LVM plus thin qcow2-on-LVM, no filesystem layer
    • -
  • VDUSE backend via qemu-storage-daemon, for shared-SAN Kubernetes
    • -
-
-
-
-
- -
-
-
-

Get started

-

Blockstor installs onto an existing cluster and is driven with the standard linstor client and piraeus linstor-csi. The usage guide walks through installing the control plane, registering nodes and pools, and wiring up CSI.

-

Read the usage guide

-
-
-

Images

-

The three images are published to GHCR on every release.

-
    -
  • ghcr.io/cozystack/blockstor-controllerthe reconcilers
    • -
  • ghcr.io/cozystack/blockstor-apiserverthe LINSTOR-compatible REST API, over mTLS
    • -
  • ghcr.io/cozystack/blockstor-satellitethe per-node DRBD and storage agent
    • -
-
-
-
- -
-
-

Documentation

-
    -
  • UsageInstall and operate blockstor with the linstor client and piraeus linstor-csi.
    • -
  • ArchitectureThe load-bearing design decisions.
    • -
  • Layer stackDRBD, LUKS and STORAGE compositions.
    • -
  • ContributingRepository layout and the local Talos and QEMU dev stand.
    • -
-
-
- -
-
-

Acknowledgements

-

Blockstor was inspired by LINBIT's LINSTOR, and it operates DRBD to provide block-level replication. It deliberately speaks LINSTOR's API so that the ecosystem the LINSTOR community has built keeps working unchanged. Heartfelt thanks to LINBIT and to the wider DRBD, LINSTOR and Piraeus community.

-
-
-
- -
-
-

Blockstor is licensed under Apache 2.0. The code is provided as-is with no warranties. Third-party Go modules and their licenses are listed in THIRD_PARTY_LICENSES.md.

-

LINSTOR, LINBIT and DRBD are trademarks or registered trademarks of LINBIT. Blockstor is an independent project and is not affiliated with, endorsed by, or sponsored by LINBIT.

-
-
- - diff --git a/site/layouts/_partials/favicons.html b/site/layouts/_partials/favicons.html new file mode 100644 index 00000000..1b4b8c8b --- /dev/null +++ b/site/layouts/_partials/favicons.html @@ -0,0 +1 @@ + diff --git a/site/package-lock.json b/site/package-lock.json new file mode 100644 index 00000000..0489bac3 --- /dev/null +++ b/site/package-lock.json @@ -0,0 +1,912 @@ +{ + "name": "site", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "devDependencies": { + "autoprefixer": "^10.4.16", + "postcss": "^8.4.32", + "postcss-cli": "^11.0.0" + } + }, + "node_modules/ansi-regex": { + "version": "5.0.1", + "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.1.tgz", + "integrity": "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=8" + } + }, + "node_modules/ansi-styles": { + "version": "4.3.0", + "resolved": "https://registry.npmjs.org/ansi-styles/-/ansi-styles-4.3.0.tgz", + "integrity": "sha512-zbB9rCJAT1rbjiVDb2hqKFHNYLxgtk8NURxZ3IZwD3F6NtxbXZQCnnSi1Lkx+IDohdPlFp222wVALIheZJQSEg==", + "dev": true, + "license": "MIT", + "dependencies": { + "color-convert": "^2.0.1" + }, + "engines": { + "node": ">=8" + }, + "funding": { + "url": "https://github.com/chalk/ansi-styles?sponsor=1" + } + }, + "node_modules/anymatch": { + "version": "3.1.3", + "resolved": "https://registry.npmjs.org/anymatch/-/anymatch-3.1.3.tgz", + "integrity": "sha512-KMReFUr0B4t+D+OBkjR3KYqvocp2XaSzO55UcB6mgQMd3KbcE+mWTyvVV7D/zsdEbNnV6acZUutkiHQXvTr1Rw==", + "dev": true, + "license": "ISC", + "dependencies": { + "normalize-path": "^3.0.0", + "picomatch": "^2.0.4" + }, + "engines": { + "node": ">= 8" + } + }, + "node_modules/autoprefixer": { + "version": "10.6.1", + "resolved": "https://registry.npmjs.org/autoprefixer/-/autoprefixer-10.6.1.tgz", + "integrity": "sha512-cL1Qz6ADZhcEbny/8HPfe99J6HhNoYtpX2LFLIbhgGE7Q1hlQVkYFdetDN7Id3KiQxhDrHwzlHr/YQCnZ8+xSA==", + "dev": true, + "funding": [ + { + "type": "opencollective", + "url": "https://opencollective.com/postcss/" + }, + { + "type": "tidelift", + "url": "https://tidelift.com/funding/github/npm/autoprefixer" + }, + { + "type": "github", + "url": "https://github.com/sponsors/ai" + } + ], + "license": "MIT", + "dependencies": { + "browserslist": "^4.28.9", + "caniuse-lite": "^1.0.30001810", + "fraction.js": "^5.3.4", + "picocolors": "^1.1.1", + "postcss-value-parser": "^4.2.0" + }, + "bin": { + "autoprefixer": "bin/autoprefixer" + }, + "engines": { + "node": "^10 || ^12 || >=14" + }, + "peerDependencies": { + "postcss": "^8.1.0" + } + }, + "node_modules/baseline-browser-mapping": { + "version": "2.11.28", + "resolved": "https://registry.npmjs.org/baseline-browser-mapping/-/baseline-browser-mapping-2.11.28.tgz", + "integrity": "sha512-br0LOmjJqr6Rt4AEkadbcNf67Zpo7EPwKsWbjnRrJIa1QYFHQBqM4C7F2WTbV8ahTEcg1Eo+Y0aWjjORInYFyA==", + "dev": true, + "license": "Apache-2.0", + "bin": { + "baseline-browser-mapping": "dist/cli.cjs" + }, + "engines": { + "node": ">=6.0.0" + } + }, + "node_modules/binary-extensions": { + "version": "2.3.0", + "resolved": "https://registry.npmjs.org/binary-extensions/-/binary-extensions-2.3.0.tgz", + "integrity": "sha512-Ceh+7ox5qe7LJuLHoY0feh3pHuUDHAcRUeyL2VYghZwfpkNIy/+8Ocg0a3UuSoYzavmylwuLWQOf3hl0jjMMIw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=8" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/braces": { + "version": "3.0.3", + "resolved": "https://registry.npmjs.org/braces/-/braces-3.0.3.tgz", + "integrity": "sha512-yQbXgO/OSZVD2IsiLlro+7Hf6Q18EJrKSEsdoMzKePKXct3gvD8oLcOQdIzGupr5Fj+EDe8gO/lxc1BzfMpxvA==", + "dev": true, + "license": "MIT", + "dependencies": { + "fill-range": "^7.1.1" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/browserslist": { + "version": "4.29.3", + "resolved": "https://registry.npmjs.org/browserslist/-/browserslist-4.29.3.tgz", + "integrity": "sha512-1R4kiYKXGViqEN0CnoDrXc1StD9niAwu+j2dukWzrD4bJgsD4lDmEp0CRbc6E/vYJIfTHwPmwyaKtVSudICdPA==", + "dev": true, + "funding": [ + { + "type": "opencollective", + "url": "https://opencollective.com/browserslist" + }, + { + "type": "tidelift", + "url": "https://tidelift.com/funding/github/npm/browserslist" + }, + { + "type": "github", + "url": "https://github.com/sponsors/ai" + } + ], + "license": "MIT", + "dependencies": { + "baseline-browser-mapping": "^2.11.26", + "caniuse-lite": "^1.0.30001813", + "electron-to-chromium": "^1.5.439", + "node-releases": "^2.0.57", + "update-browserslist-db": "^1.3.3" + }, + "bin": { + "browserslist": "cli.js" + }, + "engines": { + "node": "^6 || ^7 || ^8 || ^9 || ^10 || ^11 || ^12 || >=13.7" + } + }, + "node_modules/caniuse-lite": { + "version": "1.0.30001815", + "resolved": "https://registry.npmjs.org/caniuse-lite/-/caniuse-lite-1.0.30001815.tgz", + "integrity": "sha512-uDybUEyfgbyJJ93J1vikTylBX31AkdVYeVPeffMQpf8XN7hSfr0oZOn4VFOIMNvtU5+rHvC5Udg2TgtJKHm23Q==", + "dev": true, + "funding": [ + { + "type": "opencollective", + "url": "https://opencollective.com/browserslist" + }, + { + "type": "tidelift", + "url": "https://tidelift.com/funding/github/npm/caniuse-lite" + }, + { + "type": "github", + "url": "https://github.com/sponsors/ai" + } + ], + "license": "CC-BY-4.0" + }, + "node_modules/chokidar": { + "version": "3.6.0", + "resolved": "https://registry.npmjs.org/chokidar/-/chokidar-3.6.0.tgz", + "integrity": "sha512-7VT13fmjotKpGipCW9JEQAusEPE+Ei8nl6/g4FBAmIm0GOOLMua9NDDo/DWp0ZAxCr3cPq5ZpBqmPAQgDda2Pw==", + "dev": true, + "license": "MIT", + "dependencies": { + "anymatch": "~3.1.2", + "braces": "~3.0.2", + "glob-parent": "~5.1.2", + "is-binary-path": "~2.1.0", + "is-glob": "~4.0.1", + "normalize-path": "~3.0.0", + "readdirp": "~3.6.0" + }, + "engines": { + "node": ">= 8.10.0" + }, + "funding": { + "url": "https://paulmillr.com/funding/" + }, + "optionalDependencies": { + "fsevents": "~2.3.2" + } + }, + "node_modules/cliui": { + "version": "8.0.1", + "resolved": "https://registry.npmjs.org/cliui/-/cliui-8.0.1.tgz", + "integrity": "sha512-BSeNnyus75C4//NQ9gQt1/csTXyo/8Sb+afLAkzAptFuMsod9HFokGNudZpi/oQV73hnVK+sR+5PVRMd+Dr7YQ==", + "dev": true, + "license": "ISC", + "dependencies": { + "string-width": "^4.2.0", + "strip-ansi": "^6.0.1", + "wrap-ansi": "^7.0.0" + }, + "engines": { + "node": ">=12" + } + }, + "node_modules/color-convert": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/color-convert/-/color-convert-2.0.1.tgz", + "integrity": "sha512-RRECPsj7iu/xb5oKYcsFHSppFNnsj/52OVTRKb4zP5onXwVF3zVmmToNcOfGC+CRDpfK/U584fMg38ZHCaElKQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "color-name": "~1.1.4" + }, + "engines": { + "node": ">=7.0.0" + } + }, + "node_modules/color-name": { + "version": "1.1.4", + "resolved": "https://registry.npmjs.org/color-name/-/color-name-1.1.4.tgz", + "integrity": "sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA==", + "dev": true, + "license": "MIT" + }, + "node_modules/dependency-graph": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/dependency-graph/-/dependency-graph-1.0.0.tgz", + "integrity": "sha512-cW3gggJ28HZ/LExwxP2B++aiKxhJXMSIt9K48FOXQkm+vuG5gyatXnLsONRJdzO/7VfjDIiaOOa/bs4l464Lwg==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=4" + } + }, + "node_modules/electron-to-chromium": { + "version": "1.5.451", + "resolved": "https://registry.npmjs.org/electron-to-chromium/-/electron-to-chromium-1.5.451.tgz", + "integrity": "sha512-vUNKPrUzw8ByEMOxloq0a71iD/mPuPxVkbDm3G5YhnxTzOw/I6qAg2MUS0cYj/c267lWmw26yW4Vv7foxTxPLg==", + "dev": true, + "license": "ISC" + }, + "node_modules/emoji-regex": { + "version": "8.0.0", + "resolved": "https://registry.npmjs.org/emoji-regex/-/emoji-regex-8.0.0.tgz", + "integrity": "sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A==", + "dev": true, + "license": "MIT" + }, + "node_modules/escalade": { + "version": "3.2.0", + "resolved": "https://registry.npmjs.org/escalade/-/escalade-3.2.0.tgz", + "integrity": "sha512-WUj2qlxaQtO4g6Pq5c29GTcWGDyd8itL8zTlipgECz3JesAiiOKotd8JU6otB3PACgG6xkJUyVhboMS+bje/jA==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=6" + } + }, + "node_modules/fill-range": { + "version": "7.1.1", + "resolved": "https://registry.npmjs.org/fill-range/-/fill-range-7.1.1.tgz", + "integrity": "sha512-YsGpe3WHLK8ZYi4tWDg2Jy3ebRz2rXowDxnld4bkQB00cc/1Zw9AWnC0i9ztDJitivtQvaI9KaLyKrc+hBW0yg==", + "dev": true, + "license": "MIT", + "dependencies": { + "to-regex-range": "^5.0.1" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/fraction.js": { + "version": "5.3.4", + "resolved": "https://registry.npmjs.org/fraction.js/-/fraction.js-5.3.4.tgz", + "integrity": "sha512-1X1NTtiJphryn/uLQz3whtY6jK3fTqoE3ohKs0tT+Ujr1W59oopxmoEh7Lu5p6vBaPbgoM0bzveAW4Qi5RyWDQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": "*" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/rawify" + } + }, + "node_modules/fs-extra": { + "version": "11.4.1", + "resolved": "https://registry.npmjs.org/fs-extra/-/fs-extra-11.4.1.tgz", + "integrity": "sha512-KYAb4c9BJQI6QqGKthV68OHe0badztdXJWKo0WtBA9IuCFPTKvE5ZdUBglP833aMjhaSPNO4A5j/EkzZtGlKjA==", + "dev": true, + "license": "MIT", + "dependencies": { + "graceful-fs": "^4.2.0", + "jsonfile": "^6.0.1", + "universalify": "^2.0.0" + }, + "engines": { + "node": ">=14.14" + } + }, + "node_modules/fsevents": { + "version": "2.3.3", + "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz", + "integrity": "sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==", + "dev": true, + "hasInstallScript": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": "^8.16.0 || ^10.6.0 || >=11.0.0" + } + }, + "node_modules/get-caller-file": { + "version": "2.0.5", + "resolved": "https://registry.npmjs.org/get-caller-file/-/get-caller-file-2.0.5.tgz", + "integrity": "sha512-DyFP3BM/3YHTQOCUL/w0OZHR0lpKeGrxotcHWcqNEdnltqFwXVfhEBQ94eIo34AfQpo0rGki4cyIiftY06h2Fg==", + "dev": true, + "license": "ISC", + "engines": { + "node": "6.* || 8.* || >= 10.*" + } + }, + "node_modules/glob-parent": { + "version": "5.1.2", + "resolved": "https://registry.npmjs.org/glob-parent/-/glob-parent-5.1.2.tgz", + "integrity": "sha512-AOIgSQCepiJYwP3ARnGx+5VnTu2HBYdzbGP45eLw1vr3zB3vZLeyed1sC9hnbcOc9/SrMyM5RPQrkGz4aS9Zow==", + "dev": true, + "license": "ISC", + "dependencies": { + "is-glob": "^4.0.1" + }, + "engines": { + "node": ">= 6" + } + }, + "node_modules/graceful-fs": { + "version": "4.2.11", + "resolved": "https://registry.npmjs.org/graceful-fs/-/graceful-fs-4.2.11.tgz", + "integrity": "sha512-RbJ5/jmFcNNCcDV5o9eTnBLJ/HszWV0P73bc+Ff4nS/rJj+YaS6IGyiOL0VoBYX+l1Wrl3k63h/KrH+nhJ0XvQ==", + "dev": true, + "license": "ISC" + }, + "node_modules/is-binary-path": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/is-binary-path/-/is-binary-path-2.1.0.tgz", + "integrity": "sha512-ZMERYes6pDydyuGidse7OsHxtbI7WVeUEozgR/g7rd0xUimYNlvZRE/K2MgZTjWy725IfelLeVcEM97mmtRGXw==", + "dev": true, + "license": "MIT", + "dependencies": { + "binary-extensions": "^2.0.0" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/is-extglob": { + "version": "2.1.1", + "resolved": "https://registry.npmjs.org/is-extglob/-/is-extglob-2.1.1.tgz", + "integrity": "sha512-SbKbANkN603Vi4jEZv49LeVJMn4yGwsbzZworEoyEiutsN3nJYdbO36zfhGJ6QEDpOZIFkDtnq5JRxmvl3jsoQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/is-fullwidth-code-point": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/is-fullwidth-code-point/-/is-fullwidth-code-point-3.0.0.tgz", + "integrity": "sha512-zymm5+u+sCsSWyD9qNaejV3DFvhCKclKdizYaJUuHA83RLjb7nSuGnddCHGv0hk+KY7BMAlsWeK4Ueg6EV6XQg==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=8" + } + }, + "node_modules/is-glob": { + "version": "4.0.3", + "resolved": "https://registry.npmjs.org/is-glob/-/is-glob-4.0.3.tgz", + "integrity": "sha512-xelSayHH36ZgE7ZWhli7pW34hNbNl8Ojv5KVmkJD4hBdD3th8Tfk9vYasLM+mXWOZhFkgZfxhLSnrwRr4elSSg==", + "dev": true, + "license": "MIT", + "dependencies": { + "is-extglob": "^2.1.1" + }, + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/is-number": { + "version": "7.0.0", + "resolved": "https://registry.npmjs.org/is-number/-/is-number-7.0.0.tgz", + "integrity": "sha512-41Cifkg6e8TylSpdtTpeLVMqvSBEVzTttHvERD741+pnZ8ANv0004MRL43QKPDlK9cGvNp6NZWZUBlbGXYxxng==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=0.12.0" + } + }, + "node_modules/jsonfile": { + "version": "6.2.1", + "resolved": "https://registry.npmjs.org/jsonfile/-/jsonfile-6.2.1.tgz", + "integrity": "sha512-zwOTdL3rFQ/lRdBnntKVOX6k5cKJwEc1HdilT71BWEu7J41gXIB2MRp+vxduPSwZJPWBxEzv4yH1wYLJGUHX4Q==", + "dev": true, + "license": "MIT", + "dependencies": { + "universalify": "^2.0.0" + }, + "optionalDependencies": { + "graceful-fs": "^4.1.6" + } + }, + "node_modules/lilconfig": { + "version": "3.1.3", + "resolved": "https://registry.npmjs.org/lilconfig/-/lilconfig-3.1.3.tgz", + "integrity": "sha512-/vlFKAoH5Cgt3Ie+JLhRbwOsCQePABiU3tJ1egGvyQ+33R/vcwM2Zl2QR/LzjsBeItPt3oSVXapn+m4nQDvpzw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=14" + }, + "funding": { + "url": "https://github.com/sponsors/antonk52" + } + }, + "node_modules/nanoid": { + "version": "3.3.20", + "resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.3.20.tgz", + "integrity": "sha512-uKdg2G3GNCKQn9byYOpxbGqrT2fGO5KRt5J/8b3pok8rT6qxGWF6hxMyJiEYtAf+FVyYuD9hRaDqX5uPFYJ4ZQ==", + "dev": true, + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/ai" + } + ], + "license": "MIT", + "bin": { + "nanoid": "bin/nanoid.cjs" + }, + "engines": { + "node": "^10 || ^12 || ^13.7 || ^14 || >=15.0.1" + } + }, + "node_modules/node-releases": { + "version": "2.0.58", + "resolved": "https://registry.npmjs.org/node-releases/-/node-releases-2.0.58.tgz", + "integrity": "sha512-BdiKAgbfRlvfMq/XzO/AF9NPmaA8a+YpLAyAFTVFBj3gARrAIfWvNq7Q7NE4se3KgFu0KkrvQMxTS0XVnsGzHg==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=18" + } + }, + "node_modules/normalize-path": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/normalize-path/-/normalize-path-3.0.0.tgz", + "integrity": "sha512-6eZs5Ls3WtCisHWp9S2GUy8dqkpGi4BVSz3GaqiE6ezub0512ESztXUwUB6C6IKbQkY2Pnb/mD4WYojCRwcwLA==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/picocolors": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz", + "integrity": "sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==", + "dev": true, + "license": "ISC" + }, + "node_modules/picomatch": { + "version": "2.3.2", + "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-2.3.2.tgz", + "integrity": "sha512-V7+vQEJ06Z+c5tSye8S+nHUfI51xoXIXjHQ99cQtKUkQqqO1kO/KCJUfZXuB47h/YBlDhah2H3hdUGXn8ie0oA==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=8.6" + }, + "funding": { + "url": "https://github.com/sponsors/jonschlinkert" + } + }, + "node_modules/postcss": { + "version": "8.5.29", + "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.29.tgz", + "integrity": "sha512-49cGhUbXj8Qenv0iTMxA1cFBzxXoctpC9Ujd77t1WcbJIr6nF/eI7g/8MgxrYldFRuAXvja7xQRwavoW7kgrxQ==", + "dev": true, + "funding": [ + { + "type": "opencollective", + "url": "https://opencollective.com/postcss/" + }, + { + "type": "tidelift", + "url": "https://tidelift.com/funding/github/npm/postcss" + }, + { + "type": "github", + "url": "https://github.com/sponsors/ai" + } + ], + "license": "MIT", + "dependencies": { + "nanoid": "^3.3.19", + "picocolors": "^1.1.1", + "source-map-js": "^1.2.2" + }, + "engines": { + "node": "^10 || ^12 || >=14" + } + }, + "node_modules/postcss-cli": { + "version": "11.0.1", + "resolved": "https://registry.npmjs.org/postcss-cli/-/postcss-cli-11.0.1.tgz", + "integrity": "sha512-0UnkNPSayHKRe/tc2YGW6XnSqqOA9eqpiRMgRlV1S6HdGi16vwJBx7lviARzbV1HpQHqLLRH3o8vTcB0cLc+5g==", + "dev": true, + "license": "MIT", + "dependencies": { + "chokidar": "^3.3.0", + "dependency-graph": "^1.0.0", + "fs-extra": "^11.0.0", + "picocolors": "^1.0.0", + "postcss-load-config": "^5.0.0", + "postcss-reporter": "^7.0.0", + "pretty-hrtime": "^1.0.3", + "read-cache": "^1.0.0", + "slash": "^5.0.0", + "tinyglobby": "^0.2.12", + "yargs": "^17.0.0" + }, + "bin": { + "postcss": "index.js" + }, + "engines": { + "node": ">=18" + }, + "peerDependencies": { + "postcss": "^8.0.0" + } + }, + "node_modules/postcss-load-config": { + "version": "5.1.0", + "resolved": "https://registry.npmjs.org/postcss-load-config/-/postcss-load-config-5.1.0.tgz", + "integrity": "sha512-G5AJ+IX0aD0dygOE0yFZQ/huFFMSNneyfp0e3/bT05a8OfPC5FUoZRPfGijUdGOJNMewJiwzcHJXFafFzeKFVA==", + "dev": true, + "funding": [ + { + "type": "opencollective", + "url": "https://opencollective.com/postcss/" + }, + { + "type": "github", + "url": "https://github.com/sponsors/ai" + } + ], + "license": "MIT", + "dependencies": { + "lilconfig": "^3.1.1", + "yaml": "^2.4.2" + }, + "engines": { + "node": ">= 18" + }, + "peerDependencies": { + "jiti": ">=1.21.0", + "postcss": ">=8.0.9", + "tsx": "^4.8.1" + }, + "peerDependenciesMeta": { + "jiti": { + "optional": true + }, + "postcss": { + "optional": true + }, + "tsx": { + "optional": true + } + } + }, + "node_modules/postcss-reporter": { + "version": "7.1.0", + "resolved": "https://registry.npmjs.org/postcss-reporter/-/postcss-reporter-7.1.0.tgz", + "integrity": "sha512-/eoEylGWyy6/DOiMP5lmFRdmDKThqgn7D6hP2dXKJI/0rJSO1ADFNngZfDzxL0YAxFvws+Rtpuji1YIHj4mySA==", + "dev": true, + "funding": [ + { + "type": "opencollective", + "url": "https://opencollective.com/postcss/" + }, + { + "type": "github", + "url": "https://github.com/sponsors/ai" + } + ], + "license": "MIT", + "dependencies": { + "picocolors": "^1.0.0", + "thenby": "^1.3.4" + }, + "engines": { + "node": ">=10" + }, + "peerDependencies": { + "postcss": "^8.1.0" + } + }, + "node_modules/postcss-value-parser": { + "version": "4.2.0", + "resolved": "https://registry.npmjs.org/postcss-value-parser/-/postcss-value-parser-4.2.0.tgz", + "integrity": "sha512-1NNCs6uurfkVbeXG4S8JFT9t19m45ICnif8zWLd5oPSZ50QnwMfK+H3jv408d4jw/7Bttv5axS5IiHoLaVNHeQ==", + "dev": true, + "license": "MIT" + }, + "node_modules/pretty-hrtime": { + "version": "1.0.3", + "resolved": "https://registry.npmjs.org/pretty-hrtime/-/pretty-hrtime-1.0.3.tgz", + "integrity": "sha512-66hKPCr+72mlfiSjlEB1+45IjXSqvVAIy6mocupoww4tBFE9R9IhwwUGoI4G++Tc9Aq+2rxOt0RFU6gPcrte0A==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/read-cache": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/read-cache/-/read-cache-1.0.2.tgz", + "integrity": "sha512-/peqiBB/n07gQGLsWaHho3WfvUyRscw0gYTsEFMhrIe/nWLkYaf5SbKYjGYqtRV3aPwykJgF2VEMo1ac4bnsGA==", + "dev": true, + "license": "MIT" + }, + "node_modules/readdirp": { + "version": "3.6.0", + "resolved": "https://registry.npmjs.org/readdirp/-/readdirp-3.6.0.tgz", + "integrity": "sha512-hOS089on8RduqdbhvQ5Z37A0ESjsqz6qnRcffsMU3495FuTdqSm+7bhJ29JvIOsBDEEnan5DPu9t3To9VRlMzA==", + "dev": true, + "license": "MIT", + "dependencies": { + "picomatch": "^2.2.1" + }, + "engines": { + "node": ">=8.10.0" + } + }, + "node_modules/require-directory": { + "version": "2.1.1", + "resolved": "https://registry.npmjs.org/require-directory/-/require-directory-2.1.1.tgz", + "integrity": "sha512-fGxEI7+wsG9xrvdjsrlmL22OMTTiHRwAMroiEeMgq8gzoLC/PQr7RsRDSTLUg/bZAZtF+TVIkHc6/4RIKrui+Q==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/slash": { + "version": "5.1.0", + "resolved": "https://registry.npmjs.org/slash/-/slash-5.1.0.tgz", + "integrity": "sha512-ZA6oR3T/pEyuqwMgAKT0/hAv8oAXckzbkmR0UkUosQ+Mc4RxGoJkRmwHgHufaenlyAgE1Mxgpdcrf75y6XcnDg==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=14.16" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/source-map-js": { + "version": "1.2.2", + "resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.2.tgz", + "integrity": "sha512-KGj/8Y43x35aZVDtt+J4mK1hoLGHULMYfSkODJNQjNDC3oW1PqPoxMwo0pLUsWM/UEGzON/NxeHywEfNXNP3Vw==", + "dev": true, + "license": "BSD-3-Clause", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/string-width": { + "version": "4.2.3", + "resolved": "https://registry.npmjs.org/string-width/-/string-width-4.2.3.tgz", + "integrity": "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==", + "dev": true, + "license": "MIT", + "dependencies": { + "emoji-regex": "^8.0.0", + "is-fullwidth-code-point": "^3.0.0", + "strip-ansi": "^6.0.1" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/strip-ansi": { + "version": "6.0.1", + "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-6.0.1.tgz", + "integrity": "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==", + "dev": true, + "license": "MIT", + "dependencies": { + "ansi-regex": "^5.0.1" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/thenby": { + "version": "1.4.1", + "resolved": "https://registry.npmjs.org/thenby/-/thenby-1.4.1.tgz", + "integrity": "sha512-D5a/bO0KdalOE3q8MlrRmSxjbKZHT3MQmXkJP+r97Vw8MMwOZKOwUSEyTtK7eSMj2y0kyAjpYMRMZmmLw1FtNQ==", + "dev": true, + "license": "Apache-2.0" + }, + "node_modules/tinyglobby": { + "version": "0.2.17", + "resolved": "https://registry.npmjs.org/tinyglobby/-/tinyglobby-0.2.17.tgz", + "integrity": "sha512-wXR/dYpcqKmfWpEdZjiKJOwCNFndD0DMnrW/cYjVGttEkBfVgcLFHoNrlj47mjOVic9yyNu65alsgF4NQyTa2g==", + "dev": true, + "license": "MIT", + "dependencies": { + "fdir": "^6.5.0", + "picomatch": "^4.0.4" + }, + "engines": { + "node": ">=12.0.0" + }, + "funding": { + "url": "https://github.com/sponsors/SuperchupuDev" + } + }, + "node_modules/tinyglobby/node_modules/fdir": { + "version": "6.5.0", + "resolved": "https://registry.npmjs.org/fdir/-/fdir-6.5.0.tgz", + "integrity": "sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=12.0.0" + }, + "peerDependencies": { + "picomatch": "^3 || ^4" + }, + "peerDependenciesMeta": { + "picomatch": { + "optional": true + } + } + }, + "node_modules/tinyglobby/node_modules/picomatch": { + "version": "4.0.7", + "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.7.tgz", + "integrity": "sha512-qcJu88Q2IWqJsDD529JKMdwGm/dvInW4HvQnRwiH9JtihJvzGOscDtHE3x1pBKeUOTysQ8kVmLnJ2kJu7yhcGA==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=12" + }, + "funding": { + "url": "https://github.com/sponsors/jonschlinkert" + } + }, + "node_modules/to-regex-range": { + "version": "5.0.1", + "resolved": "https://registry.npmjs.org/to-regex-range/-/to-regex-range-5.0.1.tgz", + "integrity": "sha512-65P7iz6X5yEr1cwcgvQxbbIw7Uk3gOy5dIdtZ4rDveLqhrdJP+Li/Hx6tyK0NEb+2GCyneCMJiGqrADCSNk8sQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "is-number": "^7.0.0" + }, + "engines": { + "node": ">=8.0" + } + }, + "node_modules/universalify": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/universalify/-/universalify-2.0.1.tgz", + "integrity": "sha512-gptHNQghINnc/vTGIk0SOFGFNXw7JVrlRUtConJRlvaw6DuX0wO5Jeko9sWrMBhh+PsYAZ7oXAiOnf/UKogyiw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 10.0.0" + } + }, + "node_modules/update-browserslist-db": { + "version": "1.3.4", + "resolved": "https://registry.npmjs.org/update-browserslist-db/-/update-browserslist-db-1.3.4.tgz", + "integrity": "sha512-t8wonIOdQyOjtPd2VktlBu6QjTVTks6v8bSi7kd0r2Vll8TK1JYDZtRRdbmCHOAMphkZgrzK+L+vXK4pyZ8t4A==", + "dev": true, + "funding": [ + { + "type": "opencollective", + "url": "https://opencollective.com/browserslist" + }, + { + "type": "tidelift", + "url": "https://tidelift.com/funding/github/npm/browserslist" + }, + { + "type": "github", + "url": "https://github.com/sponsors/ai" + } + ], + "license": "MIT", + "dependencies": { + "escalade": "^3.2.0", + "picocolors": "^1.1.1" + }, + "bin": { + "update-browserslist-db": "cli.js" + }, + "peerDependencies": { + "browserslist": ">= 4.21.0" + } + }, + "node_modules/wrap-ansi": { + "version": "7.0.0", + "resolved": "https://registry.npmjs.org/wrap-ansi/-/wrap-ansi-7.0.0.tgz", + "integrity": "sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q==", + "dev": true, + "license": "MIT", + "dependencies": { + "ansi-styles": "^4.0.0", + "string-width": "^4.1.0", + "strip-ansi": "^6.0.0" + }, + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/chalk/wrap-ansi?sponsor=1" + } + }, + "node_modules/y18n": { + "version": "5.0.8", + "resolved": "https://registry.npmjs.org/y18n/-/y18n-5.0.8.tgz", + "integrity": "sha512-0pfFzegeDWJHJIAmTLRP2DwHjdF5s7jo9tuztdQxAhINCdvS+3nGINqPd00AphqJR/0LhANUS6/+7SCb98YOfA==", + "dev": true, + "license": "ISC", + "engines": { + "node": ">=10" + } + }, + "node_modules/yaml": { + "version": "2.9.1", + "resolved": "https://registry.npmjs.org/yaml/-/yaml-2.9.1.tgz", + "integrity": "sha512-3NxN8+78OdzbT7C/WjGsyfPAtJaN3FNDsWxv7Y7mcDsT/oOmgW8BpyQQFFBnvZE3j9Y2Sdz1ULFLezL7Eb2yFw==", + "dev": true, + "license": "ISC", + "bin": { + "yaml": "bin.mjs" + }, + "engines": { + "node": ">= 14.6" + }, + "funding": { + "url": "https://github.com/sponsors/eemeli" + } + }, + "node_modules/yargs": { + "version": "17.7.3", + "resolved": "https://registry.npmjs.org/yargs/-/yargs-17.7.3.tgz", + "integrity": "sha512-GZtjxm/J/4TSxuL3FNYjCmLktBTnIw/rVmKSIyKeYAZpmJB2ig9VauCC5xsa82GNKVKDAqpOn3KVzNt0zmrU0g==", + "dev": true, + "license": "MIT", + "dependencies": { + "cliui": "^8.0.1", + "escalade": "^3.1.1", + "get-caller-file": "^2.0.5", + "require-directory": "^2.1.1", + "string-width": "^4.2.3", + "y18n": "^5.0.5", + "yargs-parser": "^21.1.1" + }, + "engines": { + "node": ">=12" + } + }, + "node_modules/yargs-parser": { + "version": "21.1.1", + "resolved": "https://registry.npmjs.org/yargs-parser/-/yargs-parser-21.1.1.tgz", + "integrity": "sha512-tVpsJW7DdjecAiFpbIB1e3qxIQsE6NoPc5/eTdrbbIC4h0LVsWhnoa3g+m2HclBIujHzsxZ4VJVA+GUuc2/LBw==", + "dev": true, + "license": "ISC", + "engines": { + "node": ">=12" + } + } + } +} diff --git a/site/package.json b/site/package.json new file mode 100644 index 00000000..a190dd59 --- /dev/null +++ b/site/package.json @@ -0,0 +1,8 @@ +{ + "private": true, + "devDependencies": { + "autoprefixer": "^10.4.16", + "postcss": "^8.4.32", + "postcss-cli": "^11.0.0" + } +} diff --git a/site/img/favicon.svg b/site/static/favicon.svg similarity index 100% rename from site/img/favicon.svg rename to site/static/favicon.svg diff --git a/site/img/blockstor-logo-black.svg b/site/static/img/blockstor-logo-black.svg similarity index 100% rename from site/img/blockstor-logo-black.svg rename to site/static/img/blockstor-logo-black.svg diff --git a/site/static/img/blockstor-logo-white.svg b/site/static/img/blockstor-logo-white.svg new file mode 100644 index 00000000..1e4fb76a --- /dev/null +++ b/site/static/img/blockstor-logo-white.svg @@ -0,0 +1,6 @@ + + + + + + diff --git a/site/robots.txt b/site/static/robots.txt similarity index 100% rename from site/robots.txt rename to site/static/robots.txt diff --git a/site/style.css b/site/style.css deleted file mode 100644 index ab52cd3a..00000000 --- a/site/style.css +++ /dev/null @@ -1,518 +0,0 @@ -:root { - --paper: #f3f5f9; - --band: #e7ebf2; - --ink: #15213b; - --muted: #4a5672; - --rule: #cfd6e3; - --accent: #2f46d6; - --accent-ink: #ffffff; - --code-bg: #e2e7f0; - --block: #2f46d6; - --block-soft: #c9d1f5; - --sans: "IBM Plex Sans", system-ui, -apple-system, "Segoe UI", sans-serif; - --mono: "IBM Plex Mono", ui-monospace, SFMono-Regular, Menlo, monospace; -} - -@media (prefers-color-scheme: dark) { - :root { - --paper: #0e1524; - --band: #141d30; - --ink: #e8edf6; - --muted: #a3aec4; - --rule: #2a3651; - --accent: #93a4ff; - --accent-ink: #0e1524; - --code-bg: #1d2840; - --block: #93a4ff; - --block-soft: #2b3a66; - } -} - -*, -*::before, -*::after { - box-sizing: border-box; -} - -html { - scroll-behavior: smooth; -} - -@media (prefers-reduced-motion: reduce) { - html { - scroll-behavior: auto; - } -} - -body { - margin: 0; - background: var(--paper); - color: var(--ink); - font-family: var(--sans); - font-size: 1.0625rem; - line-height: 1.6; - -webkit-text-size-adjust: 100%; -} - -img, -svg { - max-width: 100%; - height: auto; -} - -a { - color: var(--accent); - text-underline-offset: 0.18em; -} - -a:hover { - text-decoration-thickness: 2px; -} - -:focus-visible { - outline: 3px solid var(--accent); - outline-offset: 3px; - border-radius: 2px; -} - -code { - font-family: var(--mono); - font-size: 0.88em; - background: var(--code-bg); - padding: 0.08em 0.32em; - border-radius: 3px; - overflow-wrap: anywhere; -} - -h1, -h2, -h3 { - line-height: 1.2; - margin: 0 0 0.6em; -} - -h1 { - font-size: clamp(1.9rem, 4.2vw, 3rem); - font-weight: 600; - letter-spacing: -0.015em; - max-width: 22ch; -} - -h2 { - font-size: clamp(1.5rem, 2.6vw, 2rem); - font-weight: 600; - letter-spacing: -0.01em; -} - -h3 { - font-size: 1.15rem; - font-weight: 600; -} - -p { - margin: 0 0 1em; - max-width: 68ch; -} - -.wrap { - width: 100%; - max-width: 72rem; - margin: 0 auto; - padding: 0 1rem; -} - -@media (min-width: 40rem) { - .wrap { - padding: 0 2rem; - } -} - -.narrow p { - max-width: 68ch; -} - -.skip { - position: absolute; - left: -999px; - top: 0.5rem; - background: var(--accent); - color: var(--accent-ink); - padding: 0.5rem 0.9rem; - z-index: 10; -} - -.skip:focus { - left: 0.5rem; -} - -/* Header */ - -.top { - border-bottom: 1px solid var(--rule); -} - -.top-row { - display: flex; - flex-wrap: wrap; - align-items: center; - justify-content: space-between; - gap: 0.75rem 1.5rem; - padding-top: 1rem; - padding-bottom: 1rem; -} - -.brand { - display: inline-flex; -} - -.logo { - height: 28px; - width: auto; -} - -.logo-dark { - display: none; -} - -@media (prefers-color-scheme: dark) { - .logo-light { - display: none; - } - - .logo-dark { - display: block; - } -} - -nav { - display: flex; - flex-wrap: wrap; - gap: 0.25rem 1.25rem; -} - -nav a { - color: var(--ink); - text-decoration: none; - font-weight: 500; -} - -nav a:hover { - color: var(--accent); -} - -/* Hero */ - -.hero { - padding: 3.5rem 0 4rem; -} - -.hero-grid { - display: grid; - gap: 2.5rem; - align-items: center; -} - -@media (min-width: 56rem) { - .hero-grid { - grid-template-columns: 1.15fr 1fr; - gap: 3.5rem; - } -} - -.lede { - font-size: 1.2rem; - color: var(--ink); -} - -.hero-copy p:not(.lede) { - color: var(--muted); -} - -.actions { - display: flex; - flex-wrap: wrap; - gap: 0.75rem; - margin-top: 1.75rem; -} - -.btn { - display: inline-block; - padding: 0.65rem 1.15rem; - border: 1.5px solid var(--accent); - border-radius: 4px; - color: var(--accent); - font-weight: 600; - text-decoration: none; -} - -.btn:hover { - background: var(--band); -} - -.btn-primary { - background: var(--accent); - color: var(--accent-ink); -} - -.btn-primary:hover { - background: var(--accent); - filter: brightness(1.1); -} - -/* The replication figure: the one picture on the page. */ - -.replicas { - margin: 0; -} - -.replicas svg { - display: block; - width: 100%; - max-width: 30rem; - margin: 0 auto; - font-family: var(--mono); -} - -.replicas text { - fill: var(--ink); - font-size: 13px; -} - -.replicas .sub { - fill: var(--muted); - font-size: 12px; -} - -.replicas .crd rect { - fill: var(--paper); - stroke: var(--ink); - stroke-width: 1.5; -} - -.replicas .node > rect:first-child { - fill: var(--band); - stroke: var(--rule); - stroke-width: 1.5; -} - -.replicas .blk { - fill: var(--block); -} - -.replicas .links path { - fill: none; - stroke: var(--muted); - stroke-width: 1.5; -} - -.replicas .sync path { - fill: none; - stroke: var(--block); - stroke-width: 2; - stroke-dasharray: 4 4; -} - -@media (prefers-reduced-motion: no-preference) { - .replicas .sync path { - animation: sync 1.6s linear infinite; - } -} - -@keyframes sync { - to { - stroke-dashoffset: -16; - } -} - -.replicas figcaption { - margin: 1rem auto 0; - max-width: 30rem; - color: var(--muted); - font-size: 0.95rem; -} - -/* Sections */ - -main > section { - padding: 3.5rem 0; -} - -.band { - background: var(--band); -} - -.why-grid { - display: grid; - gap: 2rem; -} - -@media (min-width: 56rem) { - .why-grid { - grid-template-columns: 1.6fr 1fr 1fr; - gap: 2.5rem; - } -} - -dl { - margin: 0; -} - -dt { - font-weight: 600; - margin-top: 1rem; -} - -dt:first-child { - margin-top: 0; -} - -dd { - margin: 0.25rem 0 0; - color: var(--muted); -} - -/* Status: the marker shape says the state, so it reads without colour. */ - -.status-grid { - display: grid; - gap: 2rem; -} - -@media (min-width: 56rem) { - .status-grid { - grid-template-columns: 1.4fr 1fr 1fr; - gap: 2.5rem; - } -} - -.status ul { - list-style: none; - margin: 0; - padding: 0; -} - -.status li { - position: relative; - padding-left: 1.6rem; - margin-bottom: 0.6rem; -} - -.status li::before { - content: ""; - position: absolute; - left: 0; - top: 0.42em; - width: 0.8rem; - height: 0.8rem; - border-radius: 2px; -} - -.works li::before { - background: var(--block); -} - -.missing li::before { - border: 1.5px solid var(--muted); -} - -.planned li::before { - border: 1.5px dashed var(--block); -} - -.note { - color: var(--muted); - font-size: 0.95rem; -} - -/* Get started */ - -.start-grid { - display: grid; - gap: 2rem; -} - -@media (min-width: 56rem) { - .start-grid { - grid-template-columns: 1fr 1fr; - gap: 3rem; - } -} - -.images { - list-style: none; - margin: 0; - padding: 0; -} - -.images li { - display: flex; - flex-direction: column; - gap: 0.15rem; - padding: 0.75rem 0; - border-top: 1px solid var(--rule); -} - -.images li:last-child { - border-bottom: 1px solid var(--rule); -} - -.images code { - background: none; - padding: 0; - font-size: 0.92rem; -} - -.images span { - color: var(--muted); - font-size: 0.95rem; -} - -/* Documentation */ - -.docs { - list-style: none; - margin: 0; - padding: 0; - display: grid; - gap: 1.25rem 2.5rem; -} - -@media (min-width: 40rem) { - .docs { - grid-template-columns: 1fr 1fr; - } -} - -.docs li { - display: flex; - flex-direction: column; - gap: 0.2rem; -} - -.docs a { - font-weight: 600; - font-size: 1.1rem; -} - -.docs span { - color: var(--muted); -} - -/* Footer */ - -.foot { - border-top: 1px solid var(--rule); - padding: 2rem 0 2.5rem; - color: var(--muted); - font-size: 0.92rem; -} - -.foot p { - max-width: 80ch; -} - -/* 404 */ - -.lost { - padding: 5rem 0; -} From 4acce6439c3f87318ec40c39aa94d3a71f9f5368 Mon Sep 17 00:00:00 2001 From: Andrei Kvapil Date: Thu, 8 Oct 2026 23:35:12 +0200 Subject: [PATCH 06/21] feat(site): rebuild the landing page with Docsy blocks Carries over the existing landing's text, which comes from the README. Assisted-by: LLM Signed-off-by: Andrei Kvapil --- site/content/en/_index.md | 73 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100644 site/content/en/_index.md diff --git a/site/content/en/_index.md b/site/content/en/_index.md new file mode 100644 index 00000000..c77090e1 --- /dev/null +++ b/site/content/en/_index.md @@ -0,0 +1,73 @@ +--- +title: Blockstor +--- + +{{< blocks/cover title="Blockstor" image_anchor="top" height="med" color="primary" >}} +

A Kubernetes control plane for LVM and ZFS storage with DRBD replication.

+Get started +GitHub +{{< /blocks/cover >}} + +{{% blocks/lead color="white" %}} +Blockstor speaks a LINSTOR-compatible REST API, so the clients you already use keep working unchanged: `linstor-csi`, `piraeus-operator`, `ha-controller`, `golinstor` and the `linstor` CLI. + +The difference is underneath. Desired state lives in CRDs, and a set of `controller-runtime` reconcilers drive the cluster toward it. There is no external database to back up, no in-memory state to lose on restart, and no controller-to-node polling to fall behind. +{{% /blocks/lead %}} + +{{% blocks/section color="light" type="row" %}} + +{{% blocks/feature icon="fa-solid fa-dharmachakra" title="Kubernetes-native" %}} +The state of truth lives in Kubernetes CRDs. `Resource`, `ResourceDefinition`, `ResourceGroup`, `StoragePool`, `Snapshot`, `Node` and `PhysicalDevice` are meant to be read, and where it makes sense written, by other operators. Each satellite watches the CRDs for its own node and reports what it observes through Status, with no gRPC dispatch from a central controller. +{{% /blocks/feature %}} + +{{% blocks/feature icon="fa-brands fa-golang" title="Ecosystem fit" %}} +Go is what the apiserver, the kubelet, etcd, most CSI drivers and `controller-runtime` are written in, so blockstor shares their tooling, libraries and contributors. +{{% /blocks/feature %}} + +{{% blocks/feature icon="fa-solid fa-feather" title="Small footprint" %}} +The controller, the apiserver and the satellite are statically linked Go binaries, with no JVM, and DRBD is a Linux kernel module rather than a userspace daemon. The images are multi-arch, `linux/amd64` and `linux/arm64`, so blockstor fits on edge and small ARM nodes. +{{% /blocks/feature %}} + +{{% /blocks/section %}} + +{{% blocks/section color="white" type="row" %}} + +{{% blocks/feature icon="fa-solid fa-check" title="Works today" %}} +- Replicated volumes over DRBD, on LVM, LVM-thin, ZFS, ZFS-thin or file backends +- Running without DRBD: plain local storage, single-replica diskful or diskless +- LUKS encryption: volume-level encryption at rest +- Autoplacement with constraints: zones, node properties, replicas on different nodes +- TieBreaker and quorum policies +- Snapshots: create, restore as a new resource, roll back, and clone +- Intra-cluster snapshot shipping (`zfs send`/`recv`, `thin-send-recv`) for clone and add-replica +- Online volume resize +- Device-pool creation from physical disks (`physical-storage create-device-pool`) +- A LINSTOR-compatible REST API for the whole client ecosystem, served over mTLS +{{% /blocks/feature %}} + +{{% blocks/feature icon="fa-solid fa-xmark" title="Not implemented" %}} +The API answers these with `501 Not Implemented`. + +- Cross-cluster snapshot shipping (disaster recovery) +- Backup create, restore, ship and abort, and the backup queue +- Schedules (cron-driven backups) +- Remote backends: S3, LINSTOR remotes +- Extra storage providers: SPDK, NVMe-oF, OpenFlex, Exos +{{% /blocks/feature %}} + +{{% blocks/feature icon="fa-solid fa-road" title="On the roadmap" %}} +- Bring-your-own-key encryption: operator-managed Secret references in the spec, instead of a controller-owned passphrase bag +- A migration tool from LINSTOR that adopts an existing LINSTOR cluster's resources into blockstor in place +- [Shared-LUN provisioning](https://github.com/oVirt/vdsm/blob/master/doc/thin-provisioning.md): thick LVM plus thin qcow2-on-LVM, no filesystem layer +- [VDUSE backend](https://blog.deckhouse.io/lvm-qcow-csi-driver-shared-san-kubernetes-81455201590e) via `qemu-storage-daemon`, for shared-SAN Kubernetes +{{% /blocks/feature %}} + +{{% /blocks/section %}} + +{{% blocks/section color="light" %}} +## Acknowledgements {.text-center} + +Blockstor was inspired by LINBIT's [LINSTOR](https://linbit.com/linstor/), and it operates [DRBD](https://linbit.com/drbd/) to provide block-level replication. It deliberately speaks LINSTOR's API so that the ecosystem the LINSTOR community has built keeps working unchanged. Heartfelt thanks to LINBIT and to the wider DRBD, LINSTOR and Piraeus community. + +Blockstor is licensed under [Apache 2.0](https://github.com/cozystack/blockstor/blob/main/LICENSE). LINSTOR, LINBIT and DRBD are trademarks or registered trademarks of LINBIT. Blockstor is an independent project and is not affiliated with, endorsed by, or sponsored by LINBIT. +{{% /blocks/section %}} From 6499ab6a2ab766a0f2d359c124cac3a06165374f Mon Sep 17 00:00:00 2001 From: Andrei Kvapil Date: Thu, 8 Oct 2026 23:35:13 +0200 Subject: [PATCH 07/21] docs(site): seed the documentation section Copies the user-facing documents from docs/ into the site, with links to repository files pointing at GitHub. The originals stay where they are until it is decided whether they move. Assisted-by: LLM Signed-off-by: Andrei Kvapil --- site/content/en/docs/_index.md | 13 + site/content/en/docs/architecture.md | 209 ++++++++++++++ site/content/en/docs/getting-started.md | 23 ++ site/content/en/docs/known-issues.md | 329 ++++++++++++++++++++++ site/content/en/docs/layer-stack.md | 118 ++++++++ site/content/en/docs/linstor-migration.md | 130 +++++++++ site/content/en/docs/usage.md | 302 ++++++++++++++++++++ 7 files changed, 1124 insertions(+) create mode 100644 site/content/en/docs/_index.md create mode 100644 site/content/en/docs/architecture.md create mode 100644 site/content/en/docs/getting-started.md create mode 100644 site/content/en/docs/known-issues.md create mode 100644 site/content/en/docs/layer-stack.md create mode 100644 site/content/en/docs/linstor-migration.md create mode 100644 site/content/en/docs/usage.md diff --git a/site/content/en/docs/_index.md b/site/content/en/docs/_index.md new file mode 100644 index 00000000..c1e450b3 --- /dev/null +++ b/site/content/en/docs/_index.md @@ -0,0 +1,13 @@ +--- +title: "Documentation" +linkTitle: "Docs" +weight: 20 +menu: + main: + weight: 10 +description: "Install, operate and understand blockstor." +--- + +Blockstor is a Kubernetes control plane for LVM and ZFS storage with DRBD replication. It speaks a LINSTOR-compatible REST API, so `linstor-csi`, `piraeus-operator`, `ha-controller`, `golinstor` and the `linstor` CLI keep working unchanged. + +Start with [Getting started](getting-started/), then see [Usage](usage/) for the full walkthrough. diff --git a/site/content/en/docs/architecture.md b/site/content/en/docs/architecture.md new file mode 100644 index 00000000..7b98d40a --- /dev/null +++ b/site/content/en/docs/architecture.md @@ -0,0 +1,209 @@ +--- +title: "Architecture" +weight: 40 +description: "The load-bearing design decisions." +--- + + + +This document captures load-bearing invariants the codebase relies +on. It is NOT a tour of the code — for that, follow the call graph +from `cmd/controller/main.go` and `cmd/satellite/main.go` +(satellite). The pieces below are the rules whose violation would +quietly corrupt cluster state. + +## Spec / Status discipline + +A blockstor CRD has two halves: + +* **Spec** is _desired state_. Operators, REST handlers, and the + controller's own placement logic write here. The satellite + **never writes Spec**. +* **Status** is _observed state_. The satellite (and the + controller's allocators that derive values from Spec) write here. + The user-facing REST API never writes Status — it returns it + read-only. + +The rule: + +> Anything the satellite reads from the kernel (`drbdsetup events2`, +> `drbdadm status`, `lvs`, …) lives on **Status**. Anything the +> operator or controller asks the satellite to do lives on **Spec**. + +A naive whole-object `Update` is unsafe whenever both halves are +written by different actors — a Spec mutation in flight would +clobber a concurrent Status write and vice versa. Status writes +**must** go through the Status subresource (`Status().Update()`); +Spec writes use the regular `Update()` path. + +This rule is enforced by code review for now; with Phase 10.2 +landed, the satellite-side reconciler's writes naturally route +through Status only, making the invariant mechanical. + +### Field placement cheatsheet + +| Field | Half | Rationale | +|---|---|---| +| `Resource.Spec.NodeName` | Spec | Operator-chosen placement target. | +| `Resource.Spec.Flags` | Spec | DISKLESS / TIE_BREAKER are operator-controlled. | +| `Resource.Spec.StoragePool` | Spec | Allocator output written by controller. | +| `Resource.Spec.Volumes[i].SeedFromGi` | Spec | Controller-picked DRBD GI to stamp on first activation. | +| `Resource.Status.InUse` | Status | Reflects `drbdsetup status` role. | +| `Resource.Status.DrbdState` | Status | Reflects `events2` connection state. | +| `Resource.Status.DRBDPort` / `DRBDMinor` / `DRBDNodeID` | Status | Allocator-derived; immutable per replica once set. | +| `Resource.Status.Volumes[i].DiskState` | Status | Per-volume kernel state. | +| `Resource.Status.Volumes[i].CurrentGi` | Status | DRBD generation identifier observed by the satellite. | +| `Node.Status.ConnectionStatus` | Status | Set on Hello — reflects whether the satellite has dialled in. | + +A field that fits neither half (rare — typically a transient +debounce hint) lives in an annotation, not Spec or Status. + +### Multi-writer Status (server-side apply) + +Some Status fields are written by **both** the controller (e.g. +allocator output → `DRBDPort`) and the satellite (e.g. +`DiskState`, `CurrentGi`). A regular `.Status().Update()` from +either side rewrites the **whole** Status subresource, which can +clobber the other side's writes that landed between Get and +Update. + +Phase 10.2 routed those writes through Kubernetes Server-Side +Apply with distinct field managers (`blockstor-controller`, +`blockstor-satellite`) so each side only touches the fields it +owns. See `pkg/store/k8s/resources.go` `SetState` for the +satellite-side writer; the controller-side writer lives in +`internal/controller`. + +## Hierarchy resolver + +DRBD configuration follows an upstream-LINSTOR-shaped override +chain: + +``` +Controller → ResourceGroup → ResourceDefinition → Resource + (broadest) (narrowest) +``` + +Lower scopes override higher scopes per non-nil field. The typed +implementation lives in `pkg/drbd/typed_resolver.go` +(`ResolveDRBDOptions`); the legacy string-keyed implementation +(`ResolveOptions`) is still used as a fallback for any +`Spec.Props` data not yet migrated to the typed `DRBDOptions` +struct. See `internal/controller/resource_controller.go`'s +`resolveEffectiveProps` for the merge. + +`*int32` and `*bool` use nil-vs-set discipline: + +* `nil` means "not overridden at this scope, inherit from parent". +* Any non-nil value (including the zero value) means "explicitly + set, do not inherit". + +A regression that did `if *src.X { out.X = src.X }` would silently +drop explicit-`false` overrides, e.g. an RD that intentionally +sets `AllowTwoPrimaries=false` would inherit a parent RG's `true`. +The pinning tests for this are in `pkg/drbd/typed_resolver_test.go`. + +## Wire format vs CRD storage + +Two shapes coexist in the codebase: + +1. **Wire shape** — `pkg/api/v1` types, identical to upstream + LINSTOR's REST API. golinstor and external callers see this + verbatim. Property bags live as `props map[string]string`. +2. **CRD shape** — `api/v1alpha1` types, the typed structures + blockstor persists in Kubernetes. DRBD configuration lives in + `Spec.DRBDOptions` (typed) + `Spec.ExtraProps` (forward-compat + for keys we haven't typed yet). + +The k8s store (`pkg/store/k8s/`) is the boundary. Its +`drbd_transcode.go` parses the wire `props` bag into typed CRD +fields on Create/Update; the inverse direction re-emits typed +fields back into `props` on GET so golinstor sees the unchanged +shape. Unknown DrbdOptions/* keys round-trip through ExtraProps +without loss. + +## DRBD initial-sync skip + +Adding a third replica to a 2-replica resource without +intervention would trigger a full resync of the entire backing +device — hours on multi-TiB volumes. The skip pipeline (Phase +8.1): + +1. Satellite's `events2` observer parses `current-uuid` from + each device frame and surfaces it in Status as + `Resource.Status.Volumes[i].CurrentGi`. +2. Controller's `ensureSeedFromGi` picks the lowest-named + UpToDate peer's CurrentGi when allocating a new replica and + stamps it on the new replica's `Spec.Volumes[i].SeedFromGi`. +3. Dispatcher threads SeedFromGi through the satellite gRPC + contract (`DesiredVolume.seed_from_gi`). +4. Satellite reconciler's `applyDRBD` runs + + ``` + drbdmeta --force / v09 internal set-gi ::0:0 + ``` + + between `drbdadm create-md` and `drbdadm adjust` on first + activation. With matching `current_uuid`+`bitmap_uuid` the + GI handshake on first connect sees the new peer as + already-in-sync and skips the full sync. + +The pipeline is end-to-end gated by +`tests/e2e/replica-add-no-resync.sh`. + +## Controllers and reconcilers + +Controller-side reconcilers (`internal/controller/`) — primary +CRD owners: + +* `ResourceReconciler` (`resource_controller.go`, ~1245 LOC) — + Resource CRDs. Allocates DRBD node-id / port / minor; picks + SeedFromGi; promotes DISKLESS to diskful when actively used; + writes desired-state to the CRD for the satellite to pick up. +* `ResourceDefinitionReconciler` + (`resourcedefinition_controller.go`, ~917 LOC) — RD CRDs. + Auto-creates `DISKLESS+TIE_BREAKER` witnesses when an RD has + even diskful replicas; sets the resource-level quorum policy. +* `NodeReconciler` (`node_controller.go`, ~239 LOC) — Node + CRDs. Owns the Node bookkeeping that the heartbeat watchdog + (Phase 10.6) relies on. +* `ResourceGroupReconciler` (`resourcegroup_controller.go`, + ~132 LOC), `StoragePoolReconciler` + (`storagepool_controller.go`, ~68 LOC), + `SnapshotReconciler` (`snapshot_controller.go`, ~60 LOC) — + smaller reconcilers that own CRD-level invariants for their + kind. The bulk of placement / capacity / snapshot-ship logic + lives in the dispatcher + satellite controllers below. + +Sibling controllers in the same package handle cross-cutting +concerns: + +* `node_heartbeat_controller.go` (~243 LOC) — Phase 10.6 + heartbeat watchdog; flips Node `ConnectionStatus` to OFFLINE + when satellites stop refreshing their lease. +* `node_label_sync_controller.go` (~326 LOC) — propagates + selected k8s Node labels to the blockstor Node CRD. +* `auto_diskful_controller.go` (~493 LOC) — DISKLESS → diskful + promotion when a resource is actively used. +* `auto_evict_controller.go` (~452 LOC) — evicts replicas from + unhealthy / drained nodes. +* `autosnapshot_controller.go` (~536 LOC) — periodic snapshot + scheduling per RG/RD policy. +* `rg_rebalance_controller.go` (~539 LOC) — rebalances RG + membership / spawns shortfall replicas. +* `resource_migration_controller.go` (~217 LOC) — migrates a + Resource between nodes. + +Satellite-side reconcilers live in `pkg/satellite/controllers/` +and watch the apiserver directly via controller-runtime (Phase +10.1). They drive DRBD / LUKS / STORAGE layers and write +observed state back via Status SSA. + +Phase 10.1 lifted the satellite's gRPC-driven reconciler logic +into those `pkg/satellite/controllers/` reconcilers. That +change retired `pkg/satellitecontroller/`, but `pkg/dispatcher/` +is still live: it provides the CRD → DesiredResource translator +(resolves layer_stack, options, passphrases) and is imported by +the satellite-side reconciler at +`pkg/satellite/controllers/resource.go` (and exercised by +`pkg/dispatcher/dispatcher_test.go`). diff --git a/site/content/en/docs/getting-started.md b/site/content/en/docs/getting-started.md new file mode 100644 index 00000000..43ba552a --- /dev/null +++ b/site/content/en/docs/getting-started.md @@ -0,0 +1,23 @@ +--- +title: "Getting started" +weight: 10 +description: "What blockstor needs and where to begin." +--- + +Blockstor installs onto an existing cluster and is driven with the standard `linstor` client and piraeus `linstor-csi`. The full walkthrough, from installing the control plane to registering nodes and pools and wiring up CSI, is in [Usage](../usage/). + +## Images + +The three images are published to GHCR on every release: + +- `ghcr.io/cozystack/blockstor-controller`: the reconcilers +- `ghcr.io/cozystack/blockstor-apiserver`: the LINSTOR-compatible REST API, over mTLS +- `ghcr.io/cozystack/blockstor-satellite`: the per-node DRBD and storage agent + +## Coming from LINSTOR + +An existing LINSTOR cluster can be moved onto blockstor; see [Migrating from LINSTOR](../linstor-migration/). + +## Contributing + +The repository layout and the local Talos and QEMU dev stand are described in [AGENTS.md](https://github.com/cozystack/blockstor/blob/main/AGENTS.md). diff --git a/site/content/en/docs/known-issues.md b/site/content/en/docs/known-issues.md new file mode 100644 index 00000000..cc6049ee --- /dev/null +++ b/site/content/en/docs/known-issues.md @@ -0,0 +1,329 @@ +--- +title: "Known issues" +weight: 60 +description: "Open problems and their workarounds." +--- + + + +This document tracks outstanding bugs and observations surfaced during the scenario/e2e validation session. Each entry references the scenario or test where it was first observed, plus the closing commit if any. + +Severity legend: + +- **P0** — data corruption, split-brain, lost volumes, or cluster-wide unavailability +- **P1** — feature unusable, retry/cancel paths missing, regression vs upstream LINSTOR +- **P2** — cosmetic, log noise, doc/RBAC drift, ergonomic gaps + +## Bug 32: force-strip ZVOL cleanup race + +**Status**: open (observation, not a defect per se) +**Severity**: P2 +**Scenario reference**: tests/scenarios/05-finalizers.md §5.34 +**Surfaced by**: scenario +**Reproduction steps**: + +1. Create a Resource with a ZFS-backed VolumeDefinition on a single node. +2. Force-delete the Resource CR (`kubectl delete --grace-period=0 --force`) while the satellite is still mid-`drbdsetup down`. +3. Observe the underlying ZVOL on the host (`zfs list -t volume`). + +**Expected behaviour**: ZVOL is destroyed in lock-step with the DRBD resource teardown; no stray dataset survives the CR removal. + +**Actual behaviour**: A short window exists where the CR is gone but the ZVOL still resides on disk because the satellite finalizer was bypassed. The next reconcile or operator restart eventually GCs it, but for a few seconds the storage pool reports phantom capacity. + +**Recommended fix**: No code change required. Document the window and rely on Bug 33 (DRBD orphan sweeper) plus Bug 43 (storage orphan sweeper) to handle the residual cleanup deterministically. + +**Related commits / tests**: see Bug 33 (DRBD sweeper) and Bug 43 (storage sweeper). + +## Bug 33: orphan kernel DRBD-Diskless after force-strip + +**Status**: closed (by Bug 5.34 sweeper) +**Severity**: P1 +**Scenario reference**: tests/scenarios/05-finalizers.md §5.34 +**Surfaced by**: scenario +**Reproduction steps**: + +1. Create a 3-replica Resource (2 disk + 1 diskless tiebreaker). +2. Force-delete the tiebreaker Resource CR before the satellite finalizer drains. +3. Inspect `drbdsetup status` on the former diskless node. + +**Expected behaviour**: Kernel DRBD device on the diskless node is fully removed; `drbdsetup status` no longer lists it. + +**Actual behaviour**: Kernel resource remained in `Diskless` state with no corresponding CR, preventing subsequent recreation on the same node (port/minor collision). + +**Recommended fix**: Periodic sweeper on the satellite enumerates kernel DRBD resources and reconciles against the local CR cache; any kernel resource without a matching CR is `drbdsetup down`-ed. + +**Related commits / tests**: 5.34 sweeper landed in the satellite controller; e2e covers via scenario 05 force-strip path. + +## Bug 34: migrate-disk endpoint missing + +**Status**: closed (by 05b8709); Option B follow-up open +**Severity**: P1 +**Scenario reference**: tests/scenarios/07-toggle-disk.md §7.4 +**Surfaced by**: e2e test +**Reproduction steps**: + +1. Create a 2-replica Resource on nodes A and B. +2. Issue `linstor resource-definition migrate-disk `. + +**Expected behaviour**: REST handler accepts the request, schedules a synchronous disk migration (toggle-disk on, then off on source) without losing redundancy. + +**Actual behaviour** (pre-fix): Endpoint returned 404; the upstream LINSTOR client could not orchestrate disk migration through the apiserver. + +**Recommended fix (landed)**: REST handler `migrate-disk` wired in 05b8709 to a two-step toggle-disk pipeline (add on target → wait UpToDate → remove on source). + +**Option B follow-up (open)**: Wrap the two-step sequence in a single reconciler-driven state machine so an apiserver restart mid-migration does not strand the user with an extra diskful replica. Track at next sprint. + +**Related commits / tests**: 05b8709; scenario 07 §7.4 covers the happy path. + +## Bug 35: placer FreeCapacity gate missing + +**Status**: open +**Severity**: P0 +**Scenario reference**: tests/scenarios/08-placer.md §8.2 +**Surfaced by**: scenario +**Reproduction steps**: + +1. Create two storage pools with vastly different free capacity (e.g. 100 GiB and 1 GiB free). +2. Create a Resource of size 10 GiB without an explicit node selector. + +**Expected behaviour**: Placer rejects the small pool because the requested size exceeds its FreeCapacity and chooses the larger pool (or fails fast with a clear error). + +**Actual behaviour**: Placer ignores FreeCapacity entirely and may select the smaller pool, leading to `ENOSPC` at ZVOL/LV create time and a stuck Resource in `Creating`. + +**Recommended fix**: Add a hard FreeCapacity gate inside the candidate-pool filter before scoring. Treat pools with `free < requested + reservedHeadroom` as ineligible. Add a placer unit test with two pools at different free sizes. + +**Related commits / tests**: none yet; blocker for placer 1.0. + +## Bug 36: handleVDUpdate wholesale-replace + +**Status**: open +**Severity**: P1 +**Scenario reference**: tests/scenarios/09-vd-update.md §9.1 +**Surfaced by**: agent +**Reproduction steps**: + +1. Create a VolumeDefinition with `props: {a: 1, b: 2}`. +2. PATCH the VD with `override_props: {a: 9}` (intent: change only `a`). + +**Expected behaviour**: Final props are `{a: 9, b: 2}` — only the keys named in `override_props` are touched. + +**Actual behaviour**: `handleVDUpdate` rewrites the entire props map from the request body, so `b` is silently dropped. + +**Recommended fix**: Replace the wholesale assignment with a merge: apply `override_props` keys on top of the existing map, then remove keys named in `delete_props`. Mirrors the upstream LINSTOR controller semantics and is the precondition for Bug 37. + +**Related commits / tests**: none; depends on Bug 37. + +## Bug 37: VolumeDefinitionModify override_props/delete_props no-op + +**Status**: open +**Severity**: P1 +**Scenario reference**: tests/scenarios/09-vd-update.md §9.2 +**Surfaced by**: e2e test +**Reproduction steps**: + +1. Apply `VolumeDefinitionModify` with `override_props: {DrbdOptions/Net/protocol: C}`. +2. GET the VD and inspect `props`. + +**Expected behaviour**: `DrbdOptions/Net/protocol` is set to `C`; `delete_props` removes named keys. + +**Actual behaviour**: Both fields are accepted by the REST schema but ignored by the handler — the props map is unchanged. + +**Recommended fix**: Wire `override_props` and `delete_props` through the same merge path introduced by Bug 36's fix. Add e2e assertions that round-trip both add and remove. + +**Related commits / tests**: depends on Bug 36; scenario 09 §9.2 has the assertion stub. + +## Bug 38: VD shrink accepted without STATE_INFO warning + +**Status**: open +**Severity**: P2 (cosmetic) +**Scenario reference**: tests/scenarios/09-vd-update.md §9.3 +**Surfaced by**: scenario +**Reproduction steps**: + +1. Create a VD of size 10 GiB. +2. PATCH the VD to size 5 GiB. + +**Expected behaviour**: Either the request is rejected, or it is accepted with a `STATE_INFO` / warning in the response indicating that shrinking is unsafe and won't reclaim space on the backing pool until the FS is also shrunk. + +**Actual behaviour**: Request is accepted silently; no warning surfaces to the operator. Backing volume keeps its original allocation. + +**Recommended fix**: Append a `STATE_INFO` entry to the API response when `newSize < currentSize`. No semantic change to storage; purely advisory. + +**Related commits / tests**: none. + +## Bug 39: toggle-disk retry counter missing + +**Status**: open +**Severity**: P1 +**Scenario reference**: tests/scenarios/07-toggle-disk.md §7.6 +**Surfaced by**: agent +**Reproduction steps**: + +1. Initiate `toggle-disk` on a Resource where the target node's storage pool is briefly unavailable. +2. Observe reconciler logs. + +**Expected behaviour**: Reconciler retries with bounded backoff and a visible retry counter on the Resource status; after N attempts it surfaces a terminal error condition. + +**Actual behaviour**: Reconciler retries forever with no visibility; if the failure is permanent, the Resource is stuck "in flight" indefinitely. + +**Recommended fix**: Add `status.toggleDisk.attempts` and `status.toggleDisk.lastError`. Cap at e.g. 10 attempts before transitioning to `Failed` with a clear condition. Couple with Bug 40. + +**Related commits / tests**: none; precondition for the migrate-disk Option B state machine. + +## Bug 40: toggle-disk cancel state machine missing + +**Status**: open +**Severity**: P1 +**Scenario reference**: tests/scenarios/07-toggle-disk.md §7.7 +**Surfaced by**: agent +**Reproduction steps**: + +1. Start a long-running `toggle-disk` (e.g. large initial sync). +2. Issue a cancel/abort via the REST API. + +**Expected behaviour**: Reconciler observes the cancel intent, rolls back the partial change (removes the half-added diskful, or re-adds the half-removed one) and returns the Resource to its pre-toggle state. + +**Actual behaviour**: No cancel endpoint exists; the only escape is to delete the Resource entirely. + +**Recommended fix**: Model toggle-disk as an explicit state machine (`Pending → Syncing → Promoting → Done | Cancelling → Rolled-back`). Add a cancel verb in the REST surface and have the reconciler honour it idempotently. + +**Related commits / tests**: none; pairs with Bug 39. + +## Bug 41: RBAC nodes get/list/watch missing in stand yaml + +**Status**: closed (13722e1) +**Severity**: P2 +**Scenario reference**: tests/scenarios/01-bootstrap.md §1.3 +**Surfaced by**: e2e test (stand bring-up) +**Reproduction steps**: + +1. Deploy blockstor on the dev stand with the previous controller-manifests yaml. +2. Watch controller logs at startup. + +**Expected behaviour**: Controller starts and lists Node objects to seed its topology cache. + +**Actual behaviour** (pre-fix): `nodes "" is forbidden: User "system:serviceaccount:..." cannot list resource "nodes" in API group "" at the cluster scope`; controller crashloops. + +**Recommended fix (landed)**: Add `nodes` verbs `get/list/watch` to the ClusterRole shipped with the stand yaml. + +**Related commits / tests**: 13722e1. + +## Bug 42: piraeus pod-CIDR drift on e2e-iptables + +**Status**: open (investigation pending) +**Severity**: P1 +**Scenario reference**: tests/scenarios/10-e2e-iptables.md §10.1 +**Surfaced by**: e2e test +**Reproduction steps**: + +1. Bring up the e2e-iptables Talos+QEMU stand via `make iter`. +2. Deploy piraeus-operator with the default values. +3. Wait for the satellite DaemonSet pods to schedule and the LINSTOR controller to register them. + +**Expected behaviour**: Satellite pod IPs come from the cluster's pod CIDR and are routable to/from the LINSTOR controller; resources reach `UpToDate` without DRBD net errors. + +**Actual behaviour**: Some satellite pods receive addresses outside the expected pod CIDR (suspected CNI bridge / kube-proxy iptables drift on the iptables-mode stand), DRBD peer connections flap, resources oscillate between `Connecting` and `Connected`. + +**Recommended fix**: Investigation pending. First steps: + +- Confirm whether the drift is at the CNI layer (bridge IPAM) or kube-proxy (SNAT/MASQUERADE on the iptables stand). +- Capture `ip a`, `iptables-save`, and `kubectl get pods -o wide` on a failing run. +- Compare against the ipvs-mode stand where the issue does not reproduce. + +**Related commits / tests**: scenario 10 §10.1 reproducer; no fix yet. + +## Bug 49: satellite ignores runtime NetInterface changes (deferred from scenario 3.10) + +**Status**: open (deferred — design gap, not a defect) +**Severity**: P2 +**Scenario reference**: tests/scenarios/03-networking.md §3.10 +**Surfaced by**: scenario audit (UG9 §"Managing network interface cards", lines 2167-2169) +**Reproduction steps**: + +1. `kubectl edit node.blockstor.io/` and re-order the entries under `spec.netInterfaces` (or change the first entry's `address`). +2. Observe the running satellite Pod on that worker — it does not re-dial anything, does not re-render its DRBD `on ` block, and continues talking to the kube-apiserver via the same Service VIP. +3. Restart the satellite Pod (`kubectl delete pod -l app=blockstor-satellite,kubernetes.io/hostname=`) — only on restart does the new `Spec.NetInterfaces[0].Address` flow into `LocalAddress` (via `$POD_IP`) and from there into the next reconcile-rendered .res file. + +**Expected behaviour (upstream LINSTOR contract)**: `linstor node interface modify ... --active` flips `StltConn/0/Active=true` and the satellite re-dials the controller via the freshly-active NIC without a process restart. + +**Actual behaviour (blockstor)**: No re-dial happens, and there is no field to flip. Two architectural reasons: + +1. Phase 10.6 retired the satellite→controller gRPC wire entirely. The satellite now talks to the kube-apiserver via the standard in-cluster config (`ctrl.GetConfig()` → `kubernetes.default.svc` Service VIP). There is no LINSTOR-style "controller endpoint" left to re-target. +2. `api/v1alpha1.NodeNetInterface` has no `Active` field. The `IsActive` flag on the REST wire is synthesised at conversion time (`i == 0` in `pkg/store/k8s/nodes.go`) — pure presentation, no behaviour. The only satellite code that reads peer NetInterfaces at runtime is `pkg/satellite/controllers/snapshot_fetcher.go` (peer-to-peer snapshot transport), and it picks by `Name == "default"` or first non-empty `Address`, not by an Active flag. + +**Recommended fix**: Deferred. Implementing Outcome A (live NIC switch) would require: + +- Adding an `Active` field to `NodeNetInterface` and a controller-runtime watch in the satellite that triggers reconciliation when `Spec.NetInterfaces[].Active` changes. +- A redirect plane that the satellite actually re-routes through — but the satellite's only outbound wire is the kube-apiserver REST/watch, which is owned by client-go and routes via the cluster Service VIP rather than this node's NIC selection. +- For the satellite→satellite snapshot stream the change would be marginal (peer-side already picks Address dynamically per Fetch call). + +Net: implementing the upstream contract requires either resurrecting a custom controller-bind wire (rejected by the Phase 10.6 design) or accepting that "active interface" is now a pure presentation field that operators must not rely on for connection re-targeting. + +The spec is pinned by `TestSatelliteFlagsLackControllerBindAddress` and `TestNodeCRDHasNoActiveField` in `pkg/satellite/stream/redial_spec_test.go`. If a future change implements Outcome A, both tests will fail and force the reviewer to replace them with a positive re-dial assertion. + +**Related commits / tests**: `pkg/satellite/stream/redial_spec_test.go`; tests/scenarios/03-networking.md §3.10. + +## Bug 43: mass-churn ZVOL leak on force-stripped finalizers + +**Status**: closed (by storage-orphan sweeper) +**Severity**: P0 +**Scenario reference**: tests/e2e/lc-rd-delete-churn.sh (scenario 5.4 extended — 10 RD rapid create/delete on one node) +**Surfaced by**: e2e test + +**Reproduction steps**: + +1. Run `lc-rd-delete-churn.sh` with `ITERS=10` against a 2-node stand. +2. Inject a satellite stall (e.g. SIGSTOP on the satellite pod) for the iteration's DRBD-down phase so the REST controller times out the per-iter delete and force-strips the Resource CRD finalizer. +3. After the loop, `zfs list -t volume` on each node. + +**Expected behaviour**: All 10 ZVOLs are destroyed in lock-step with the Resource CRD deletion; the final `zfs list -t volume` shows no `churn-N_NNNNN` datasets. + +**Actual behaviour** (pre-fix): 9 of 10 ZVOLs leak. The REST controller stamps `DeletionTimestamp` on the per-node Resource CRD; the satellite finalizer is mid-`drbdsetup down` so doesn't release the finalizer in time; the controller force-strips the finalizer; the satellite's `handleDelete` (which would have called `provider.DeleteVolume`) never runs; the ZVOL survives indefinitely. Bug 32 documents the short-window variant; Bug 43 is the deterministic mass-churn variant where the satellite genuinely cannot keep up and force-strip becomes the steady-state path. + +**Recommended fix (landed)**: New `StorageOrphanSweeperRunnable` in `pkg/satellite/controllers/storage_sweeper.go` periodically (every 5 minutes, same cadence as the DRBD orphan sweeper for Bug 33) walks every registered `storage.Provider` that implements the new `storage.VolumeLister` capability, diffs the on-disk volume list against the Resource CRDs scheduled to this node, and calls `provider.DeleteVolume` on each volume whose owning CRD has fully vanished (force-strip aftermath). + +Safety belts on the new path: + +- Wildcard-owned protection: a Resource CRD whose RD has been cascade-deleted out of order still counts as "owned" (volume number wildcard) so the sweeper never races `handleDelete`. +- Per-cycle delete bound (`StorageSweeperMaxDeletePerCycle = 3`) so a pathological upstream producer of orphans surfaces in logs without being silently cleaned in one tick. +- Storage-resource prefix allowlist (`pvc-`, `bs-`) so operator-created datasets that happen to share the pool are immune to the sweep. +- Per-Node opt-out annotation `blockstor.io/skip-orphan-storage-sweeper` for forensic-mode operators. +- `ListVolumeNames` skips ZFS deferred-delete markers (`__DELETED__`) and LVM snapshot LVs via the lv_attr column so the sweeper never confuses cleanup artefacts for orphans. + +Idempotency of `provider.DeleteVolume` was already guaranteed by Bug 33's contract; the sweeper relies on that — racing the satellite's own `handleDelete` is harmless because the second call returns nil. + +**Related commits / tests**: `pkg/satellite/controllers/storage_sweeper.go` + `pkg/satellite/controllers/storage_sweeper_test.go` (7 unit tests covering owned-leave-alone, orphan-reap, foreign-prefix immunity, per-node scope, rate-limit, skip-annotation, mid-delete-race protection, missing-RD wildcard protection); `pkg/storage/storage.go` adds the `VolumeLister` optional interface + `VolumeRef`; ZFS / LVM-thin / LVM-thick providers implement it. E2e validation via `lc-rd-delete-churn.sh` (passes 10 iters without ZVOL leak). + +## Bug 50: concurrent rapid late `vd c` wedges or drops the second volume + +**Status**: closed (#164 converge + #168 resize-deadlock guard; v0.1.14) (campaign tracking id: BUG-048) +**Severity**: P1 (availability; NOT data-loss, NOT a node-reboot deadlock; recoverable) +**Scenario reference**: tests/e2e/cli-matrix/multi-volume-late-vd-create.sh +**Surfaced by**: release-gate validation campaign +**Reproduction steps**: + +1. Create a multi-replica ResourceDefinition and let it converge (at least one VolumeDefinition, all replicas UpToDate). +2. Issue two `linstor volume-definition create ` calls back-to-back in rapid succession (concurrently, or scripted with no wait between them). + +**Expected behaviour**: Both volume-definitions are created; each new volume is laid down on every replica and reaches `UpToDate` (with one replica electing SyncSource), exactly as a sequential add would. + +**Actual behaviour**: Intermittently one of two faults occurs — (a) the second `vd c` returns SUCCESS but the VolumeDefinition is silently dropped (only one VD ends up persisted), or (b) the second volume is created but wedges `Inconsistent` with no SyncSource elected, so it never reaches `UpToDate`. In case (a) both calls report success. + +**Scope / reachability**: NOT reachable through the production CSI path. linstor-csi maps one PVC to one ResourceDefinition with exactly one VolumeDefinition, created in a single request; autoplace and `ControllerPublishVolume` add no VolumeDefinitions, and resize grows the existing volume in place. The fault is reachable only when an operator (or a script) issues concurrent / rapid back-to-back manual `vd c` calls against an existing multi-replica RD. No data is lost or corrupted (md5 held across all validation runs) and no node reboot is required. + +**Recovery**: Delete the wedged or missing volume-definition (`linstor vd d `) and re-add it sequentially — issue one `vd c`, wait for it to reach `UpToDate`, then issue the next. Always add late volume-definitions one at a time. + +**Recommended fix**: Serialise concurrent VolumeDefinition creation on a given ResourceDefinition (optimistic-concurrency precondition on the RD volume-minor allocation plus the auto-numbered VD create) so the second add observes the first, and seed the new volume's GI / current-UUID before peers attach so a deterministic SyncSource is elected. A first attempt (PR #164) reduced the silent-drop and fixed a day0 split-brain leg but regressed the CSI-reachable single-late-VD path, so it was not merged; a corrected fix that does not touch the single-VD path is tracked. + +**Related commits / tests**: cli-matrix cell `tests/e2e/cli-matrix/multi-volume-late-vd-create.sh` honestly catches this (as of #162); PR #164 (deferred). No closing commit yet. + +## Recommended next-fix order + +1. **Bug 35** (P0, placer FreeCapacity) — only P0; blocks placer 1.0 and risks ENOSPC at create time. +2. **Bug 42** (P1, piraeus pod-CIDR drift) — blocks the iptables-mode e2e lane. +3. **Bug 36 + 37** (P1, VD props merge) — fix together; 37 depends on 36's merge plumbing. +4. **Bug 39 + 40** (P1, toggle-disk retry/cancel) — fix together; together they unlock Bug 34's Option B state machine. +5. **Bug 50** (FIXED in v0.1.14, #164+#168) — serialise VolumeDefinition create on the RD and seed GI before peer attach; operator-only path, recoverable, not data-loss. Closed by #164 (converge) + #168 (resize-deadlock guard). +6. **Bug 34 Option B** (P1 follow-up) — wrap migrate-disk in the new state machine once 39/40 land. +7. **Bug 38** (P2, cosmetic STATE_INFO on shrink) — pure UX, do last. +8. **Bug 32** (P2, observation) — document only; no code fix needed. diff --git a/site/content/en/docs/layer-stack.md b/site/content/en/docs/layer-stack.md new file mode 100644 index 00000000..426c0302 --- /dev/null +++ b/site/content/en/docs/layer-stack.md @@ -0,0 +1,118 @@ +--- +title: "Layer stack" +weight: 50 +description: "DRBD, LUKS and STORAGE compositions." +--- + + + +blockstor mirrors LINSTOR's `layer_list` model: a `ResourceDefinition` +declares an ordered chain of layers, each adding capabilities on top +of the layer below. The satellite walks the chain bottom-up when +provisioning and top-down when tearing down. + +## Layers + +| Layer | What it does | +|-----------|-----------------------------------------------------------------------| +| `STORAGE` | Allocates the raw block device (LVM-thin LV, ZFS volume, loopfile, …) | +| `LUKS` | `cryptsetup luksFormat` + `luksOpen` over the storage device | +| `DRBD` | Renders `.res` and runs `drbdadm` to replicate over the network | + +The first entry in `layerStack` is the topmost layer the consumer Pod +mounts. The last entry is always `STORAGE` (the storage layer is +required — every replica needs a backing device, even diskless ones +where it's the network leg). + +## Common compositions + +| Stack | Use case | +|-----------------------------|---------------------------------------------------------------------| +| `["DRBD","STORAGE"]` | Default. Replicated PVC, the cozystack production case. | +| `["LUKS","STORAGE"]` | Single-replica encrypted PVC. No DRBD overhead. | +| `["DRBD","LUKS","STORAGE"]` | Encrypted at-rest + replicated. Per-volume cipher. | +| `["STORAGE"]` | Single-replica local mode. Ephemeral cache, scratch, ZFS dataset. | + +The default is `["DRBD","STORAGE"]` — when `RD.Spec.LayerStack` is +empty the controller inherits from the parent +`ResourceGroup.Spec.SelectFilter.LayerStack`, and when both are empty +the dispatcher falls through to the satellite's default-DRBD path. + +## Setting the stack + +### Per-RD via REST + +```bash +curl -XPOST http://controller:3370/v1/resource-definitions \ + -H 'Content-Type: application/json' \ + -d '{"resource_definition":{"name":"pvc-1","layer_stack":["LUKS","STORAGE"]}}' +``` + +### Per-RG via spawn template + +```bash +linstor rg create encrypted-rg +linstor rg sp encrypted-rg DrbdOptions/Encryption/passphrase '<32-byte-secret>' +linstor rg c --place-count 2 encrypted-rg +linstor rg spawn encrypted-rg pvc-2 1G --layer-list LUKS DRBD +``` + +(blockstor accepts `linstor`'s `--layer-list` flag verbatim because +the REST shape mirrors upstream LINSTOR.) + +### Per-RD via kubectl + +```yaml +apiVersion: blockstor.cozystack.io/v1alpha1 +kind: ResourceDefinition +metadata: {name: pvc-encrypted} +spec: + layerStack: ["LUKS", "STORAGE"] + props: + DrbdOptions/Encryption/passphrase: "32-byte-secret" + volumeDefinitions: + - {volumeNumber: 0, sizeKib: 1048576} +``` + +## LUKS specifics + +- Passphrase is per-RD via `DrbdOptions/Encryption/passphrase`. The + upstream `linstor rd set-property` key. +- The dispatcher folds the resolver-resolved passphrase onto the wire + as `DesiredResource.Props["LuksPassphrase"]`; the satellite reads it + from there. +- Empty passphrase with `LUKS` in the stack fails the apply rather + than silently producing an unencrypted volume. +- Mapper name: `--luks` → `/dev/mapper/--luks`. Stable + across reconciles so reopen on satellite restart re-uses it. +- Volume grow: the satellite runs `cryptsetup resize` on the mapper + after the storage layer has resized the underlying LV, before DRBD + resizes the replicated device. + +## What blockstor doesn't yet support + +- **Pluggable layer ordering**: the stack must be one of the four + rows in the table above. Arbitrary orderings (e.g. `["LUKS","DRBD","STORAGE"]` + with LUKS-over-DRBD instead of DRBD-over-LUKS) aren't yet validated + or rendered. +- **Per-volume LUKS keys**: every volume on a 2-volume RD currently + uses the same RD-level passphrase. Per-volume keys (with master-key + wrapping in the controller's KV store) is a follow-up. +- **Mid-stack changes**: editing `layerStack` after a Resource is + active doesn't re-encrypt or unwrap existing data. Treat it as + set-once-per-RD until migration support lands. +- **Cluster-passphrase rotation**: the cluster passphrase + (`POST /v1/encryption/passphrase`) wraps RD passphrases in the + controller's KV store; rotating it doesn't re-encrypt LUKS headers. + Rotating the per-RD passphrase requires `cryptsetup luksChangeKey` + which isn't wired yet — operators should drop the RD and recreate. + +## Testing + +- `pkg/api/v1/layer_stack_test.go` — RD → RG → default resolution. +- `pkg/satellite/reconciler_drbd_test.go::TestApplySkipsDRBDWhenLayerStackOmits` + — `["STORAGE"]` produces no `.res` and no drbdadm. +- `pkg/satellite/reconciler_drbd_test.go::TestApplyLayersLUKS` — pin + cryptsetup luksFormat + luksOpen run on first activation. +- `pkg/satellite/reconciler_drbd_test.go::TestApplyLUKSFailsWithoutPassphrase` + — pin the explicit-error path. diff --git a/site/content/en/docs/linstor-migration.md b/site/content/en/docs/linstor-migration.md new file mode 100644 index 00000000..418f9c1d --- /dev/null +++ b/site/content/en/docs/linstor-migration.md @@ -0,0 +1,130 @@ +--- +title: "Migrating from LINSTOR" +weight: 30 +description: "Move an existing LINSTOR cluster onto blockstor." +--- + + + +`linstor-migrate` converts a LINSTOR k8s-backend database dump into blockstor CRDs. blockstor then **adopts** the existing on-disk data (zvols/LVs + DRBD metadata) in place — no data is copied or resynced. This runbook is the required procedure; the converter is only one step of it. + +> **Scope.** This adopts a LINSTOR deployment whose satellites already run on the nodes blockstor will manage, keeping the same storage pools, DRBD minors, node-ids and (with the port capture below) TCP ports. It does not move data between nodes and does not change replica placement. + +## What the converter does and does not migrate + +Migrated: nodes, storage pools, resource groups, resource definitions (+ volume definitions), resources (replicas), and SUCCESSFUL snapshots. Controller-wide `DrbdOptions/*` become a `ControllerConfig`. DRBD minors, per-replica node-ids, the shared-secret and (when captured) TCP ports are preserved verbatim. An RD is stamped `Initialized=true` whenever its data still exists somewhere, and every replica gets `skipInitialSync=false`, so any replica added **after** migration performs a real sync instead of falsely coming up UpToDate. The latch is conditional, and it errs towards staying on: it is dropped only where no data-bearing replica is being adopted. That covers two shapes — every replica already DELETE-flagged, so nothing is migrating at all, and every replica diskless, so replicas are migrating but a client or quorum witness carries no copy to latch on. A definition whose replicas were held back because their nodes did not migrate keeps the latch, because that data is still on those nodes; so does one whose replicas were skipped for pool divergence. Unlatching in either of those would let the controller seed an empty first sync over data that exists. The report on stderr names every definition in these states, and names each reason that applies rather than only the first, so do not clear a latch by hand on the strength of a definition arriving without replicas. Snapshots are stamped `blockstor.io/adopted` so the controller records them as complete without re-taking them. + +**NOT migrated — read the report on stderr for the exact list per cluster:** + +- **LUKS passphrases.** LUKS-encrypted volumes convert with their layer stack intact, but the per-volume passphrase (encrypted with the LINSTOR master key) is not decrypted. You must provision `spec.encryption` for those RDs by hand before the encrypted volumes can be opened. Each affected RD is reported. +- **Backup-shipping remotes**, unknown/runtime flag bits, and DELETE'd / FAILED_DEPLOYMENT rows are skipped and reported. None affect live data availability. +- **DRBD TCP ports** are absent from most LINSTOR 1.33 dumps — see "Capture live DRBD ports" below. + +## Pre-flight (do all of these before touching anything) + +1. **Cluster is healthy.** `linstor r l --faulty` is empty; every resource is `UpToDate` on its diskful replicas; no node is `OFFLINE`. Do not migrate a degraded cluster — an Inconsistent/SyncTarget replica adopted as-is stays that way. + +2. **Take the database dump** (LINSTOR's k8s-backend CRDs): + + ```bash + mkdir dump && cd dump + kubectl get crds | grep -o ".*.internal.linstor.linbit.com" \ + | xargs -I{} sh -c 'kubectl get {} -ojson > {}.json' + cd .. + ``` + +3. **Capture the live DRBD ports.** LINSTOR 1.33 does not persist the port in its database, so the dump alone cannot preserve it. If blockstor adopts with a different port, `drbdadm adjust` moves the connection endpoint under live I/O (a reconnect blip, and on a marginal quorum a transient quorum loss). Capture the running ports from every satellite and union them: + + ```bash + for pod in $(kubectl -n cozy-linstor get pods -l app.kubernetes.io/component=linstor-satellite \ + -o jsonpath='{.items[*].metadata.name}'); do + kubectl -n cozy-linstor exec "$pod" -- sh -c ' + for f in /var/lib/linstor.d/*.res; do + rd=$(basename "$f" .res) + port=$(grep -oE "address[^;]*:[0-9]+" "$f" | grep -oE "[0-9]+$" | head -1) + [ -n "$port" ] && echo "$rd $port" + done' + done | sort -u > drbd-ports.txt + ``` + + Every replica of an RD shares one port, so duplicates across nodes are expected; `sort -u` collapses them. If an RD appears with two different ports, stop and investigate before proceeding. + +4. **Back up the ZFS/LVM pools' snapshots** (optional but recommended): a `zfs snapshot -r` of each pool gives an instant rollback point that is independent of both control planes. + +5. **Verify the zvol names blockstor will adopt match what is on disk.** Adoption is name-based: blockstor addresses each volume as `/_` (e.g. `data/pvc-abc…_00000`), and `CreateVolume` is idempotent only when that dataset already exists. A byte mismatch (ZFS names are case-sensitive) makes blockstor create a fresh EMPTY zvol next to the real one. Multi-replica DRBD would self-heal by SyncTarget, but **single-replica `["STORAGE"]` volumes have no resync fallback** — a mismatch there silently presents an empty disk. Cross-check before cutover, read-only, on each satellite: + + ```bash + # what is on disk: + zfs list -H -o name -t volume | sort > /tmp/ondisk-zvols.txt + # what blockstor will look for (from the converted manifests): + # /_ + # compare the two sets; every single-replica STORAGE-only volume MUST + # already exist on disk under the blockstor name. + ``` + + If any single-replica volume's computed name is absent from `ondisk-zvols.txt`, STOP — do not cut over until the naming is reconciled (`zfs rename`, or fix the converter). + + **Adopted snapshots are addressed the same way** — `/_00000@` — so run the same cross-check for them, otherwise a name miss only surfaces when someone tries to restore, long after cutover: + + ```bash + zfs list -H -o name -t snapshot | sort > /tmp/ondisk-snapshots.txt + # every migrated Snapshot's /_00000@ must appear here + ``` + +6. **Rehearse on staging first.** Do a full dry-run adoption of a COPY of the pools (or a representative subset) on a non-production cluster: apply the manifests, confirm every StoragePool registers a provider (no `provider requires … in props` errors in the satellite log), every Resource reaches `UpToDate` with `out-of-sync=0` (adopted, not resynced), and no new empty zvols appear. Only after a clean staging rehearsal proceed to production. + +## Convert + +```bash +linstor-migrate -in ./dump -drbd-ports ./drbd-ports.txt -out ./blockstor-manifests.yaml +``` + +Read the report on stderr end to end. Every `warning:` line is a thing the converter refused to guess — decide per line whether it matters for your data before continuing. In particular resolve every `LUKS passphrase NOT migrated` and every `no DRBD port` line (the latter should be empty if the port capture was complete). + +## Cutover + +The manifests do not migrate a cluster by themselves — the control plane must be swapped. Perform this in a maintenance window. + +1. **Freeze provisioning.** Scale the CSI provisioner/attacher to zero (or cordon new PVC creation) so no new volumes are created against the old controller mid-cutover. + +2. **Stop the LINSTOR controller.** Scale `deploy/linstor-controller` to 0. The satellites and the running DRBD devices keep serving I/O — only the control plane goes away. Do NOT stop the satellites or unload DRBD. + +3. **Deploy blockstor** (apiserver + controller + satellites) pointed at the same nodes and pools. The satellites must land on the same nodes with the same pool backing devices. Do not let blockstor create-md or mkfs — the adoption guards (existing DRBD metadata is not re-created, a populated filesystem is not re-formatted) protect the data, but verify the satellite image is the intended version first. + +4. **Apply the manifests in order.** `WriteManifests` already emits them in dependency order, so a single apply works: + + ```bash + kubectl apply -f ./blockstor-manifests.yaml + ``` + + If you prefer staged application, the order is: `ControllerConfig` → `Node` → `StoragePool` → `ResourceGroup` → `ResourceDefinition` → `Resource` → `Snapshot`. + +5. **Wait for adoption to converge.** Every `Resource` should report its DRBD device `UpToDate` (adopted, not resynced — watch that `out-of-sync` stays 0, i.e. no unexpected full sync started). `linstor r l --faulty` against the blockstor apiserver is empty. Storage pools report their real free capacity. + +6. **Repoint and unfreeze CSI.** Point linstor-csi at the blockstor apiserver, then scale the provisioner/attacher back up. Provision one test PVC and confirm it binds and that an existing PVC still mounts read-write. + +## Verify (before declaring done) + +- Object counts match the source: `linstor n l`, `sp l`, `rg l`, `rd l`, `r l` against blockstor equal the pre-cutover counts (the converter's report prints them). +- Spot-check ≥5 RDs: size, per-volume `/dev/drbd`, per-replica node-id and port match the pre-migration values. A changed minor or node-id is a stop-the-line defect. +- Every diskful replica is `UpToDate`; no replica is unexpectedly `SyncTarget`. +- Adopted snapshots list as present without any suspend-io having fired on the parent RDs (check the satellites did not log `suspend-io`). +- For LUKS RDs: their volumes stay closed until you provision `spec.encryption`; confirm consumers of those volumes are quiesced until then. + +## Rollback + +Until CSI is repointed and confirmed, rollback is: scale blockstor to 0, scale `deploy/linstor-controller` back to 1, repoint CSI at LINSTOR. The on-disk data and running DRBD devices were never touched, so this is a control-plane-only revert. If a pool-level `zfs snapshot` was taken in pre-flight, it is the last-resort data rollback point. + +## Known limitations (accept or mitigate explicitly) + +| Item | Impact | Mitigation | +| --- | --- | --- | +| LUKS passphrases not migrated | Encrypted volumes cannot be opened until `spec.encryption` is provisioned | Provision by hand; keep those consumers quiesced during migration | +| DRBD port not in dump (LINSTOR 1.33) | Adoption reconnects the mesh on a new port if not captured | Capture with the pre-flight command and pass `-drbd-ports` | +| Unknown flag bits dropped | Cosmetic runtime bookkeeping only | Reported; verified not to carry availability semantics | +| Placement/data unchanged | The converter never moves data | Rebalance with blockstor after migration if desired | +| `Initialized` is conditional | Left unlatched only where no data-bearing replica is adopted — every replica DELETE-flagged, or every replica diskless; a definition held back by an unmigrated node or by pool divergence keeps the latch and arrives short of replicas | Read the report, which names every reason that applies. Do not clear a latch by hand because a definition has no replicas — that is what re-enables the auto-primary election this latch exists to suppress | +| Single-replica zvol name must byte-match on disk | A mismatch presents an empty disk (no DRBD resync fallback) | Pre-flight step 5 cross-check + staging rehearsal (step 6) | +| Snapshots cover volume 0 only | blockstor addresses a snapshot as `/_00000@`; a snapshot that captured a multi-volume resource adopts with only its first volume restorable | Reported per snapshot by the converter; re-take those snapshots after migration if the other volumes matter | +| Undecoded flag bits are reported, not interpreted | A `non-zero vlm_flags` / `node_flags` / `unhandled flags bits` line means the source cluster had a marker this tool refuses to guess at (e.g. a size-semantics or eviction bit) | Look the object up in the source cluster before cutover (`linstor v l` / `n l`) and confirm it should be adopted as-is; the volume's `SizeKib` is carried verbatim either way | +| Node type mapping is empirically calibrated | Only `node_type=2` (SATELLITE) was confirmed against a live cluster; a CONTROLLER/AUXILIARY ordering error would adopt a controller node as AUXILIARY instead of skipping it | Controller nodes carry no pools or replicas, so impact is bounded — but check the report for unexpected node kinds | diff --git a/site/content/en/docs/usage.md b/site/content/en/docs/usage.md new file mode 100644 index 00000000..4293c96a --- /dev/null +++ b/site/content/en/docs/usage.md @@ -0,0 +1,302 @@ +--- +title: "Usage" +weight: 20 +description: "Install blockstor on a cluster and drive it with the linstor client and piraeus linstor-csi." +--- + + + +This guide covers deploying blockstor onto a Kubernetes cluster and driving it with the upstream `linstor` command-line client. blockstor exposes a LINSTOR-compatible REST API, so the stock `linstor` client (`python-linstor` / `linstor-client`) works unchanged. + +If you only want a throwaway local dev environment, use the Talos+QEMU stand instead (`make up && make blockstor`, see the [README](https://github.com/cozystack/blockstor/blob/main/README.md) Quick start). This page is for installing onto a real cluster. + +## Components + +blockstor ships three container images, published by CI on every version tag (see [`.github/workflows/release.yml`](https://github.com/cozystack/blockstor/blob/main/.github/workflows/release.yml)): + +| Image | Dockerfile target | Role | +|-------|-------------------|------| +| `ghcr.io/cozystack/blockstor` | `controller` | controller-runtime manager hosting the RD / RG / RP / Snapshot / Resource / Node reconcilers. Runs as a Deployment. | +| `ghcr.io/cozystack/blockstor-apiserver` | `apiserver` | Stateless LINSTOR-compatible REST front end backed by the CRD store. Runs as a multi-replica Deployment behind a Service. This is what the `linstor` client, linstor-csi and piraeus-operator talk to. | +| `ghcr.io/cozystack/blockstor-satellite` | `satellite` | Per-node manager that reconciles the DRBD / LUKS / STORAGE layers and shells out to `drbdadm` / `lvs` / `zfs` / `cryptsetup`. Runs as a DaemonSet on storage nodes. | + +Each tag publishes `MAJOR.MINOR.PATCH`, `MAJOR.MINOR`, and (for stable, non pre-release tags) `latest`. + +## Host requirements + +Storage nodes that run the satellite need: + +- The DRBD 9 kernel module loaded (`modprobe drbd`). +- `drbd-utils`, `lvm2`, `cryptsetup`, and (for ZFS pools) the ZFS kernel module plus `zfsutils-linux`. The satellite image ships the userspace tools; the kernel modules come from the host (for Talos, via the `siderolabs/drbd` and `siderolabs/zfs` extensions). +- At least one spare block device or a pre-created LVM volume group / ZFS pool to back the storage pools. + +## Deploy + +The reference manifests live under `config/crd/bases/` (CRDs) and `stand/` (the controller / apiserver / satellite workloads). Apply in this order: + +1. **CRDs** — `Resource`, `ResourceDefinition`, `ResourceGroup`, `StoragePool`, `Snapshot`, `Node`, `PhysicalDevice`, `ControllerConfig`: + + ```sh + kubectl apply -f config/crd/bases/ + ``` + +2. **Node CRs** — one cluster-scoped `Node` per storage node, with `metadata.name` equal to the Kubernetes node name. These can be created up front (the satellites also reconcile against them) or registered later through the `linstor` client (see below). A minimal Node CR: + + ```yaml + apiVersion: blockstor.cozystack.io/v1alpha1 + kind: Node + metadata: + name: worker-1 + spec: + type: SATELLITE + netInterfaces: + - name: default + address: 10.0.0.11 # the node's InternalIP, used for DRBD replication + ``` + +3. **Workloads** — the controller, apiserver, and satellite DaemonSet. The manifests in `stand/` carry an `__REGISTRY__/:dev` placeholder for the dev stand; for a real cluster set the `image:` fields to the published tags, e.g. `ghcr.io/cozystack/blockstor-controller:`, `ghcr.io/cozystack/blockstor-apiserver:`, `ghcr.io/cozystack/blockstor-satellite:`: + + - `stand/blockstor-deploy.yaml` — controller + RBAC + the `blockstor-system` namespace. + - `stand/blockstor-apiserver-deploy.yaml` — apiserver Deployment + Service. + - `stand/blockstor-satellite-daemonset.yaml` — satellite DaemonSet. + +All workloads run in the `blockstor-system` namespace. Wait for them to come up: + +```sh +kubectl -n blockstor-system rollout status deploy/blockstor-controller +kubectl -n blockstor-system rollout status deploy/blockstor-apiserver +kubectl -n blockstor-system rollout status daemonset/blockstor-satellite +``` + +## Point the `linstor` client at the apiserver + +Install the client (`apt install linstor-client`, or `pip install python-linstor`). The apiserver Service exposes a mutual-TLS listener on port **3371**; a plain-HTTP debug port **3370** is bound pod-local only. The simplest way to reach it from a workstation is a port-forward to the plain-HTTP port: + +```sh +kubectl -n blockstor-system port-forward deploy/blockstor-apiserver 3370:3370 +``` + +Then point the client at it (set this once per shell): + +```sh +export LS_CONTROLLERS=http://localhost:3370 +# or pass --controllers on every call: +linstor --controllers http://localhost:3370 node list +``` + +For in-cluster clients (linstor-csi, piraeus-operator) use the mTLS Service endpoint on 3371 with the issued client certificate instead of the port-forward. + +A quick connectivity check: + +```sh +linstor controller version +linstor node list +``` + +> Note: `linstor controller version` reports the blockstor version with a `git=blockstor` build stamp (e.g. `1.33.2+ git=blockstor`) rather than a hex commit hash — this is intentional. See [`docs/cli-parity-known-deltas.md`](https://github.com/cozystack/blockstor/blob/main/docs/cli-parity-known-deltas.md) for the full list of user-visible divergences from upstream LINSTOR (notably `advise`, `backup`, and `schedule` subcommands are not implemented). + +## Basic setup, end to end + +The commands below assume `LS_CONTROLLERS` is exported (otherwise add `--controllers …`). Replace node names, devices, and pool names with your own. + +### 1. Register nodes + +If you did not pre-create the `Node` CRs, register each satellite node: + +```sh +linstor node create worker-1 10.0.0.11 +linstor node create worker-2 10.0.0.12 +linstor node create worker-3 10.0.0.13 +linstor node list +``` + +The IP is the address used for DRBD replication traffic between nodes. Each node automatically gets a `DfltDisklessStorPool` (diskless) pool so it can host diskless replicas and tiebreakers. + +### 2. Create a storage pool from a physical device + +The one-shot way is `physical-storage create-device-pool` (short: `ps cdp`), which prepares the device and registers the pool in a single call. For a ZFS pool on `/dev/sdb`: + +```sh +linstor physical-storage create-device-pool \ + --pool-name data --storage-pool data zfs worker-1 /dev/sdb +``` + +`zfs` selects the provider (also `lvm`, `lvmthin`, `zfsthin`). Repeat per node, or pass multiple devices to build a multi-device pool. If you already have a volume group or ZFS pool, register it directly instead: + +```sh +linstor storage-pool create lvmthin worker-1 data my_vg/my_thinpool +``` + +Check the pools (and their free capacity) once the satellite has reported back: + +```sh +linstor storage-pool list +``` + +### 3. Define a resource group (recommended) + +A resource group captures replication and placement policy once, so each volume spawned from it inherits the settings: + +```sh +linstor resource-group create mygroup --place-count 3 --storage-pool data +linstor volume-group create mygroup +linstor resource-group list +``` + +### 4. Create a replicated volume + +There are two equivalent paths. + +**One-shot via the resource group** — creates the resource-definition, volume-definition, replicas, and tiebreaker in a single call: + +```sh +linstor resource-group spawn mygroup myvolume 10G +``` + +**Explicit, step by step** — useful when you are not using a resource group: + +```sh +linstor resource-definition create myvolume +linstor volume-definition create myvolume 10G +linstor resource create myvolume --auto-place 3 --storage-pool data +``` + +`--auto-place N` lets the autoplacer pick N nodes (capacity-weighted, anti-affinity aware). To pin replicas to specific nodes instead, name them: + +```sh +linstor resource create worker-1 worker-2 myvolume --storage-pool data +``` + +### 5. Inspect state + +```sh +linstor node list # nodes and their online status +linstor storage-pool list # pools, provider, free/total capacity +linstor resource-definition list # resource definitions and their groups +linstor resource list # per-node replica placement and DRBD state +linstor volume list # per-volume size and device path +``` + +A healthy 3-way replicated volume shows two `UpToDate` diskful replicas plus one `TieBreaker` (diskless) row in `linstor resource list`, all sharing the same DRBD port. + +## Dynamic provisioning: piraeus components + the `linstor-csi` driver + +So far you have driven blockstor by hand with the `linstor` client. To let Kubernetes provision DRBD-replicated PVs on demand, point the LINBIT CSI driver (`linstor-csi`) at the blockstor apiserver and create a `StorageClass`. The CSI driver speaks the same LINSTOR REST API the `linstor` client does, so blockstor is a drop-in controller backend for it. + +### What blockstor replaces — deploy the client side only + +blockstor **is** the LINSTOR controller plus the satellites: the controller binary hosts the reconcilers, the apiserver serves the REST API, and the satellite DaemonSet drives DRBD on each node. piraeus is normally a full LINSTOR distribution (operator + `linstor-controller` + `linstor-satellite` + the CSI driver). When running against blockstor you take only the **client-side** pieces from piraeus and skip everything blockstor already provides: + +| piraeus component | Against blockstor | +|-------------------|-------------------| +| `linstor-controller` | **Skip** — blockstor's controller + apiserver already are the controller. | +| `linstor-satellite` | **Skip** — blockstor ships its own satellite DaemonSet (`stand/blockstor-satellite-daemonset.yaml`). | +| `linstor-csi` (csi-controller + csi-node) | **Deploy** — this is what provisions PVs. Point it at the blockstor apiserver. | +| HA / affinity controller, `drbd-reactor` | Optional — deploy if you want failover/affinity. Point them at the apiserver the same way as the CSI driver (mTLS :3371 + client cert). | + +> The dev stand's `make piraeus` (`stand/install-piraeus.sh`) installs piraeus in **external-controller mode** against the blockstor apiserver — exactly the wiring described below. The `LinstorCluster` is created with `spec.externalController.url: https://blockstor-apiserver.blockstor-system.svc:3371`, which disables piraeus's bundled in-cluster `linstor-controller` Deployment and re-renders `linstor-csi` with `LS_CONTROLLERS` pointing at blockstor. Use that script as a working reference for a blockstor-backed cluster. + +### Point `linstor-csi` at the blockstor apiserver + +The CSI driver's controller endpoint is the apiserver's mTLS Service on port **3371** — the `blockstor-apiserver` Service (a legacy `blockstor-controller` Service name resolves to the same apiserver pods and is also covered by the serving cert's SANs, for backward compatibility with older `LS_CONTROLLERS` values). The plain-HTTP debug port 3370 is pod-local and is not an option here. `linstor-csi` wraps `golinstor`, which upgrades the endpoint to HTTPS and presents a client certificate when the `LS_USER_*` / `LS_ROOT_CA` variables point at PEM files. The exact wiring (verbatim from `stand/csi-sanity-job.yaml`, the in-repo reference for a CSI client against blockstor): + +```yaml +image: quay.io/piraeusdatastore/piraeus-csi:v1.10.1 +args: + - --csi-endpoint=$(CSI_ENDPOINT) + - --node=$(KUBE_NODE_NAME) # downward-API spec.nodeName; must match a blockstor Node name + - --linstor-endpoint=$(LS_CONTROLLERS) +env: + - {name: CSI_ENDPOINT, value: "unix:///csi/csi.sock"} + - name: KUBE_NODE_NAME + valueFrom: {fieldRef: {fieldPath: spec.nodeName}} + # mTLS endpoint — the only port the Service exposes. + - {name: LS_CONTROLLERS, value: "https://blockstor-apiserver.blockstor-system.svc:3371"} + - {name: LS_USER_CERTIFICATE, value: "/etc/linstor/client/tls.crt"} + - {name: LS_USER_KEY, value: "/etc/linstor/client/tls.key"} + - {name: LS_ROOT_CA, value: "/etc/linstor/client/ca.crt"} +volumeMounts: + - {name: client-tls, mountPath: /etc/linstor/client, readOnly: true} +volumes: + - name: client-tls + secret: + secretName: blockstor-apiserver-client-tls +``` + +Apply the same `LS_CONTROLLERS` / `LS_USER_*` / `LS_ROOT_CA` env and the same `client-tls` mount to **both** the csi-controller Deployment and the csi-node DaemonSet. If you deploy `linstor-csi` through the piraeus operator instead of plain manifests, set the cluster's external controller URL to `https://blockstor-apiserver.blockstor-system.svc:3371` (the field is `LinstorCluster.spec.externalController.url`) and wire `spec.apiTLS.certManager` at the `blockstor-api-ca` Issuer so the operator issues and mounts the client cert onto the CSI pods for you. `stand/install-piraeus.sh` is the in-repo reference for that wiring. + +### mTLS client certificate via cert-manager + +The apiserver runs a `RequireAndVerifyClientCert` TLS listener and verifies every client cert against the CA it was issued from, so the CSI driver needs a client cert from that same CA. `stand/blockstor-apiserver-tls.yaml` provisions the whole chain with cert-manager: + +``` +ca-bootstrapper (self-signed Issuer) + └─> blockstor-api-ca (CA Certificate, 10y) + └─> blockstor-api-ca (CA Issuer) + ├─> blockstor-apiserver-server (serving cert; Service DNS SANs) → Secret blockstor-apiserver-server-tls + └─> blockstor-apiserver-client (client cert for in-cluster consumers) → Secret blockstor-apiserver-client-tls +``` + +The `blockstor-apiserver-client` Certificate emits Secret `blockstor-apiserver-client-tls` carrying `tls.crt`, `tls.key` **and** `ca.crt`. Mount that Secret into the CSI pods (as above): `tls.crt`/`tls.key` authenticate the client to the apiserver, and `ca.crt` lets the client verify the apiserver's serving cert. cert-manager rotates these in place and the apiserver hot-reloads them without a restart, so no reloader annotations are needed. The serving cert's SANs cover both `blockstor-apiserver` and the legacy `blockstor-controller` Service names, so either FQDN validates cleanly. + +If the CSI driver runs in a different namespace from the apiserver, replicate the client Secret there (e.g. issue a second `Certificate` from the same `blockstor-api-ca` Issuer into that namespace) — a Secret is namespace-scoped and the Issuer must be reachable. + +### Create a StorageClass and a PVC + +Use the `linstor.csi.linbit.com` provisioner. The parameter keys are the standard `linstor.csi.linbit.com/*`-prefixed CSI keys; the values map onto blockstor's resource-group / storage-pool / placement semantics. The shape below mirrors `tests/smoke.sh`: + +```yaml +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: blockstor-replicated +provisioner: linstor.csi.linbit.com +parameters: + linstor.csi.linbit.com/storagePool: "data" # the blockstor storage pool to place diskful replicas in + linstor.csi.linbit.com/placementCount: "2" # number of diskful replicas (autoplacer adds a tiebreaker) + # Optional — pin volumes to a resource group you pre-created with `linstor resource-group create`. + # The default RG is the canonical "DfltRscGrp". + # linstor.csi.linbit.com/resourceGroup: "mygroup" +volumeBindingMode: WaitForFirstConsumer +allowVolumeExpansion: true +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: blockstor-test +spec: + accessModes: [ReadWriteOnce] + storageClassName: blockstor-replicated + resources: + requests: + storage: 1Gi +``` + +`storagePool` and `placementCount` are the keys exercised by blockstor's own smoke and csi-sanity tests, and `resourceGroup` is honored verbatim (the default RG is `DfltRscGrp`). Other `linstor.csi.linbit.com/*` keys that upstream `linstor-csi` supports — e.g. `layerList`, `disklessStoragePool`, `allowRemoteVolumeAccess` — are passed straight through to the LINSTOR REST API; verify the ones you rely on against your deployment before depending on them. + +### Verify + +```sh +kubectl get pvc blockstor-test # PVC should bind once a consumer schedules it +kubectl get pv # the bound PV's name is the LINSTOR resource name (pvc-) +``` + +Then confirm the resource shows up through the `linstor` client (port-forward or in-cluster, as above): + +```sh +linstor resource-definition list # a pvc- RD appears +linstor resource list # placementCount diskful replicas (UpToDate) + a TieBreaker, sharing one DRBD port +linstor volume list # the backing volume and its device path +``` + +A bound PVC plus an `UpToDate` resource in `linstor resource list` confirms the CSI driver is provisioning against blockstor end to end. + +## Cleanup + +Deleting the resource-definition removes every replica, frees the DRBD port, and tears down the backing volumes: + +```sh +linstor resource-definition delete myvolume +``` + +For CSI-provisioned volumes, delete the PVC instead (`kubectl delete pvc blockstor-test`) — the CSI driver issues the resource-definition delete for you. From 701baf67e15f0026261f6f43578f3280efa7c519 Mon Sep 17 00:00:00 2001 From: Andrei Kvapil Date: Thu, 8 Oct 2026 23:35:13 +0200 Subject: [PATCH 08/21] ci(site): build Netlify deploy previews with Hugo Assisted-by: LLM Signed-off-by: Andrei Kvapil --- netlify.toml | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/netlify.toml b/netlify.toml index dba41573..b5352b50 100644 --- a/netlify.toml +++ b/netlify.toml @@ -1,12 +1,13 @@ # Netlify builds deploy previews of the blockstor.org site for pull # requests. Production is GitHub Pages (.github/workflows/pages.yml). # -# The site is plain static files under site/, so there is no build command: -# Netlify publishes the directory as it is. +# The site is Hugo with the Docsy theme under site/; base moves the build +# there, so every path below is relative to site/. [build] base = "site" - publish = "." + publish = "public" + command = "npm install && hugo --gc --minify" # Skip the preview when nothing under site/ or this file changed. The # command runs from the base directory and exit 0 skips the build, so a # missing or identical cached ref forces one. @@ -15,6 +16,21 @@ git diff --quiet "$CACHED_COMMIT_REF" "$COMMIT_REF" -- . ../netlify.toml ''' +[build.environment] + HUGO_VERSION = "0.164.0" + HUGO_EXTENDED = "true" + NODE_VERSION = "22" + GO_VERSION = "1.23" + +[context.production.environment] + HUGO_ENV = "production" + +[context.deploy-preview] + command = "npm install && hugo --gc --minify -b $DEPLOY_PRIME_URL" + +[context.branch-deploy] + command = "npm install && hugo --gc --minify -b $DEPLOY_PRIME_URL" + [[redirects]] from = "/*" to = "/404.html" From d2aac3f626719e32bdd90f3dc6812cacf01a54e1 Mon Sep 17 00:00:00 2001 From: Andrei Kvapil Date: Thu, 8 Oct 2026 23:35:13 +0200 Subject: [PATCH 09/21] ci(site): build the Pages deployment with Hugo Assisted-by: LLM Signed-off-by: Andrei Kvapil --- .github/workflows/pages.yml | 40 ++++++++++++++++++++++++++++++++++--- 1 file changed, 37 insertions(+), 3 deletions(-) diff --git a/.github/workflows/pages.yml b/.github/workflows/pages.yml index c06a2ac3..4d54f85e 100644 --- a/.github/workflows/pages.yml +++ b/.github/workflows/pages.yml @@ -1,7 +1,7 @@ name: Deploy site to Pages -# blockstor.org is the static site under site/. It is published as is, with -# no build step; Netlify builds the deploy previews (netlify.toml). +# blockstor.org is the Hugo and Docsy site under site/. Netlify builds the +# deploy previews from the same sources (netlify.toml). on: push: branches: [main] @@ -24,19 +24,53 @@ concurrency: jobs: build: runs-on: ubuntu-latest + env: + HUGO_VERSION: 0.164.0 steps: - name: Checkout uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 with: persist-credentials: false + - name: Install Hugo + run: | + wget -q -O "${RUNNER_TEMP}/hugo.deb" \ + "https://github.com/gohugoio/hugo/releases/download/v${HUGO_VERSION}/hugo_extended_${HUGO_VERSION}_linux-amd64.deb" + sudo dpkg -i "${RUNNER_TEMP}/hugo.deb" + + - name: Setup Go + uses: actions/setup-go@4b73464bb391d4059bd26b0524d20df3927bd417 # v6.3.0 + with: + go-version: "1.23" + cache: false + + - name: Setup Node.js + uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0 + with: + # Hugo runs Node asset tools with "node --permission", which needs + # Node 22.13 or later. + node-version: "22" + - name: Setup Pages + id: pages uses: actions/configure-pages@45bfe0192ca1faeb007ade9deae92b16b8254a0d # v6.0.0 + - name: Install Node.js dependencies + working-directory: site + run: npm ci + + - name: Build with Hugo + working-directory: site + env: + HUGO_ENVIRONMENT: production + HUGO_ENV: production + BASE_URL: ${{ steps.pages.outputs.base_url }} + run: hugo --gc --minify --baseURL "${BASE_URL}/" + - name: Upload artifact uses: actions/upload-pages-artifact@fc324d3547104276b827a68afc52ff2a11cc49c9 # v5.0.0 with: - path: ./site + path: ./site/public deploy: environment: From ed70e6e02a5869da15287007febb727b753778f7 Mon Sep 17 00:00:00 2001 From: Andrei Kvapil Date: Thu, 8 Oct 2026 23:38:14 +0200 Subject: [PATCH 10/21] feat(site): bring the replication figure back to the cover The animated figure of one ResourceDefinition reconciled into three DRBD replicas was the one picture on the first landing page and got lost in the move to Docsy. It is a shortcode now, drawn in light strokes for the cover background, and still honours reduced motion. Assisted-by: LLM Signed-off-by: Andrei Kvapil --- site/assets/scss/_styles_project.scss | 70 ++++++++++++++++++++++++++ site/content/en/_index.md | 12 ++++- site/layouts/_shortcodes/replicas.html | 29 +++++++++++ 3 files changed, 109 insertions(+), 2 deletions(-) create mode 100644 site/layouts/_shortcodes/replicas.html diff --git a/site/assets/scss/_styles_project.scss b/site/assets/scss/_styles_project.scss index edd2e46a..e9e7277c 100644 --- a/site/assets/scss/_styles_project.scss +++ b/site/assets/scss/_styles_project.scss @@ -7,3 +7,73 @@ height: 1.75rem; width: auto; } + +// The replication figure on the cover. It sits on the primary background, so +// it is drawn in light strokes in both themes. +.replicas { + margin: 1rem auto 2rem; + max-width: 30rem; + + svg { + display: block; + width: 100%; + font-family: $font-family-monospace; + } + + text { + fill: #fff; + font-size: 13px; + } + + .sub { + fill: rgba(255, 255, 255, 0.7); + font-size: 12px; + } + + .crd rect { + fill: rgba(255, 255, 255, 0.08); + stroke: #fff; + stroke-width: 1.5; + } + + .node > rect:first-child { + fill: rgba(255, 255, 255, 0.06); + stroke: rgba(255, 255, 255, 0.35); + stroke-width: 1.5; + } + + .blk { + fill: #93a4ff; + } + + .links path { + fill: none; + stroke: rgba(255, 255, 255, 0.6); + stroke-width: 1.5; + } + + .sync path { + fill: none; + stroke: #93a4ff; + stroke-width: 2; + stroke-dasharray: 4 4; + } + + figcaption { + margin-top: 1rem; + color: rgba(255, 255, 255, 0.75); + font-size: 0.95rem; + } +} + +@media (prefers-reduced-motion: no-preference) { + .replicas .sync path { + animation: replicas-sync 1.6s linear infinite; + } +} + +@keyframes replicas-sync { + to { + stroke-dashoffset: -16; + } +} diff --git a/site/content/en/_index.md b/site/content/en/_index.md index c77090e1..014ad5d9 100644 --- a/site/content/en/_index.md +++ b/site/content/en/_index.md @@ -2,10 +2,18 @@ title: Blockstor --- -{{< blocks/cover title="Blockstor" image_anchor="top" height="med" color="primary" >}} -

A Kubernetes control plane for LVM and ZFS storage with DRBD replication.

+{{< blocks/cover image_anchor="top" height="auto" color="primary" >}} +
+
+

Blockstor

+

A Kubernetes control plane for LVM and ZFS storage with DRBD replication.

Get started GitHub +
+
+{{< replicas >}} +
+
{{< /blocks/cover >}} {{% blocks/lead color="white" %}} diff --git a/site/layouts/_shortcodes/replicas.html b/site/layouts/_shortcodes/replicas.html new file mode 100644 index 00000000..14839cdf --- /dev/null +++ b/site/layouts/_shortcodes/replicas.html @@ -0,0 +1,29 @@ +
+ + One resource definition in the Kubernetes API, reconciled into three DRBD replicas on three nodes + + + ResourceDefinition + + + + + + + {{- range $i, $x := slice 12 132 252 }} + + + node-{{ add $i 1 }} + + + + satellite + + {{- end }} + + + + + +
Each satellite watches its own slice of the CRDs and writes what it observes back through Status, while DRBD keeps the replicas in sync.
+
From 7238222f15d2d025000929a88b5b316773234421 Mon Sep 17 00:00:00 2001 From: Andrei Kvapil Date: Thu, 8 Oct 2026 23:42:21 +0200 Subject: [PATCH 11/21] feat(site): say on the landing page that blockstor is part of Cozystack Blockstor is meant to stand on its own, but it is developed within Cozystack, and the landing page should say so. The section carries the CNCF Sandbox line and logo the way cozystack.io does. Assisted-by: LLM Signed-off-by: Andrei Kvapil --- site/assets/scss/_styles_project.scss | 11 ++++ site/content/en/_index.md | 2 + site/layouts/_shortcodes/cncf.html | 10 +++ site/static/img/logos/cncf-color.svg | 88 +++++++++++++++++++++++++++ 4 files changed, 111 insertions(+) create mode 100644 site/layouts/_shortcodes/cncf.html create mode 100644 site/static/img/logos/cncf-color.svg diff --git a/site/assets/scss/_styles_project.scss b/site/assets/scss/_styles_project.scss index e9e7277c..85347faa 100644 --- a/site/assets/scss/_styles_project.scss +++ b/site/assets/scss/_styles_project.scss @@ -77,3 +77,14 @@ stroke-dashoffset: -16; } } + +.cncf-block .container { + padding-top: 4rem; + padding-bottom: 3rem; +} + +.cncf-logo { + width: 50%; + max-width: 36rem; + height: auto; +} diff --git a/site/content/en/_index.md b/site/content/en/_index.md index 014ad5d9..c36fa397 100644 --- a/site/content/en/_index.md +++ b/site/content/en/_index.md @@ -72,6 +72,8 @@ The API answers these with `501 Not Implemented`. {{% /blocks/section %}} +{{< cncf >}} + {{% blocks/section color="light" %}} ## Acknowledgements {.text-center} diff --git a/site/layouts/_shortcodes/cncf.html b/site/layouts/_shortcodes/cncf.html new file mode 100644 index 00000000..02bf236c --- /dev/null +++ b/site/layouts/_shortcodes/cncf.html @@ -0,0 +1,10 @@ +
+
+
+

Part of Cozystack

+

Blockstor is built to become an independent project, but today it is developed as part of Cozystack.

+

Cozystack is a Cloud Native Computing Foundation Sandbox project

+ +
+
+
diff --git a/site/static/img/logos/cncf-color.svg b/site/static/img/logos/cncf-color.svg new file mode 100644 index 00000000..6ed42883 --- /dev/null +++ b/site/static/img/logos/cncf-color.svg @@ -0,0 +1,88 @@ + + + + + + + + + + + + + + + + + + + + + + From f0febbf5e267cf8512dc8796704ef5f77b413c1e Mon Sep 17 00:00:00 2001 From: Andrei Kvapil Date: Thu, 8 Oct 2026 23:43:54 +0200 Subject: [PATCH 12/21] feat(site): drop the status lists from the landing page The works-today, not-implemented and roadmap lists go stale with every release and belong in the docs rather than on the front page. Assisted-by: LLM Signed-off-by: Andrei Kvapil --- site/content/en/_index.md | 34 ---------------------------------- 1 file changed, 34 deletions(-) diff --git a/site/content/en/_index.md b/site/content/en/_index.md index c36fa397..cadb539f 100644 --- a/site/content/en/_index.md +++ b/site/content/en/_index.md @@ -38,40 +38,6 @@ The controller, the apiserver and the satellite are statically linked Go binarie {{% /blocks/section %}} -{{% blocks/section color="white" type="row" %}} - -{{% blocks/feature icon="fa-solid fa-check" title="Works today" %}} -- Replicated volumes over DRBD, on LVM, LVM-thin, ZFS, ZFS-thin or file backends -- Running without DRBD: plain local storage, single-replica diskful or diskless -- LUKS encryption: volume-level encryption at rest -- Autoplacement with constraints: zones, node properties, replicas on different nodes -- TieBreaker and quorum policies -- Snapshots: create, restore as a new resource, roll back, and clone -- Intra-cluster snapshot shipping (`zfs send`/`recv`, `thin-send-recv`) for clone and add-replica -- Online volume resize -- Device-pool creation from physical disks (`physical-storage create-device-pool`) -- A LINSTOR-compatible REST API for the whole client ecosystem, served over mTLS -{{% /blocks/feature %}} - -{{% blocks/feature icon="fa-solid fa-xmark" title="Not implemented" %}} -The API answers these with `501 Not Implemented`. - -- Cross-cluster snapshot shipping (disaster recovery) -- Backup create, restore, ship and abort, and the backup queue -- Schedules (cron-driven backups) -- Remote backends: S3, LINSTOR remotes -- Extra storage providers: SPDK, NVMe-oF, OpenFlex, Exos -{{% /blocks/feature %}} - -{{% blocks/feature icon="fa-solid fa-road" title="On the roadmap" %}} -- Bring-your-own-key encryption: operator-managed Secret references in the spec, instead of a controller-owned passphrase bag -- A migration tool from LINSTOR that adopts an existing LINSTOR cluster's resources into blockstor in place -- [Shared-LUN provisioning](https://github.com/oVirt/vdsm/blob/master/doc/thin-provisioning.md): thick LVM plus thin qcow2-on-LVM, no filesystem layer -- [VDUSE backend](https://blog.deckhouse.io/lvm-qcow-csi-driver-shared-san-kubernetes-81455201590e) via `qemu-storage-daemon`, for shared-SAN Kubernetes -{{% /blocks/feature %}} - -{{% /blocks/section %}} - {{< cncf >}} {{% blocks/section color="light" %}} From 2fd81d8222e0baaa6dde8be3ad3777c161c283a8 Mon Sep 17 00:00:00 2001 From: Andrei Kvapil Date: Thu, 8 Oct 2026 23:44:18 +0200 Subject: [PATCH 13/21] feat(site): drop the heading over the Cozystack section The two lines and the CNCF logo say it without one. Assisted-by: LLM Signed-off-by: Andrei Kvapil --- site/layouts/_shortcodes/cncf.html | 1 - 1 file changed, 1 deletion(-) diff --git a/site/layouts/_shortcodes/cncf.html b/site/layouts/_shortcodes/cncf.html index 02bf236c..8a9d3f92 100644 --- a/site/layouts/_shortcodes/cncf.html +++ b/site/layouts/_shortcodes/cncf.html @@ -1,7 +1,6 @@
-

Part of Cozystack

Blockstor is built to become an independent project, but today it is developed as part of Cozystack.

Cozystack is a Cloud Native Computing Foundation Sandbox project

From f0cff1b7612addc9f3f1dae09746937e6ef90262 Mon Sep 17 00:00:00 2001 From: Andrei Kvapil Date: Thu, 8 Oct 2026 23:45:27 +0200 Subject: [PATCH 14/21] feat(site): show the white CNCF logo in the dark theme The colour logo is unreadable on the dark background. The white one is the official variant from cncf/artwork. Assisted-by: LLM Signed-off-by: Andrei Kvapil --- site/assets/scss/_styles_project.scss | 20 ++++++++++++++++++++ site/layouts/_shortcodes/cncf.html | 3 ++- site/static/img/logos/cncf-white.svg | 1 + 3 files changed, 23 insertions(+), 1 deletion(-) create mode 100644 site/static/img/logos/cncf-white.svg diff --git a/site/assets/scss/_styles_project.scss b/site/assets/scss/_styles_project.scss index 85347faa..d5ec6bf5 100644 --- a/site/assets/scss/_styles_project.scss +++ b/site/assets/scss/_styles_project.scss @@ -88,3 +88,23 @@ max-width: 36rem; height: auto; } + +// Docsy sets data-bs-theme on ; the white CNCF logo replaces the colour +// one on the dark background. +.cncf-logo--light { + display: block; +} + +.cncf-logo--dark { + display: none; +} + +[data-bs-theme="dark"] { + .cncf-logo--light { + display: none; + } + + .cncf-logo--dark { + display: block; + } +} diff --git a/site/layouts/_shortcodes/cncf.html b/site/layouts/_shortcodes/cncf.html index 8a9d3f92..a4e9560b 100644 --- a/site/layouts/_shortcodes/cncf.html +++ b/site/layouts/_shortcodes/cncf.html @@ -3,7 +3,8 @@

Blockstor is built to become an independent project, but today it is developed as part of Cozystack.

Cozystack is a Cloud Native Computing Foundation Sandbox project

- + +
diff --git a/site/static/img/logos/cncf-white.svg b/site/static/img/logos/cncf-white.svg new file mode 100644 index 00000000..d94aaf32 --- /dev/null +++ b/site/static/img/logos/cncf-white.svg @@ -0,0 +1 @@ + \ No newline at end of file From e4dac6eb62814c8bc9a00628faebe8c6f8b6e6cc Mon Sep 17 00:00:00 2001 From: Andrei Kvapil Date: Thu, 8 Oct 2026 23:46:32 +0200 Subject: [PATCH 15/21] feat(site): carry the LF copyright and policies in the footer Blockstor is developed within Cozystack, so the footer matches cozystack.io: the LF Projects copyright line and the pointer to the LF project policies. Assisted-by: LLM Signed-off-by: Andrei Kvapil --- site/hugo.yaml | 4 +--- site/layouts/_partials/footer/center.html | 2 ++ 2 files changed, 3 insertions(+), 3 deletions(-) create mode 100644 site/layouts/_partials/footer/center.html diff --git a/site/hugo.yaml b/site/hugo.yaml index f55062f8..ff0c8f1a 100644 --- a/site/hugo.yaml +++ b/site/hugo.yaml @@ -45,9 +45,7 @@ menu: params: description: A Kubernetes control plane for LVM and ZFS storage with DRBD replication. - copyright: - authors: The Blockstor Authors - from_year: 2026 + copyright: Cozystack a Series of LF Projects, LLC github_repo: https://github.com/cozystack/blockstor github_subdir: site github_branch: main diff --git a/site/layouts/_partials/footer/center.html b/site/layouts/_partials/footer/center.html new file mode 100644 index 00000000..da885b6d --- /dev/null +++ b/site/layouts/_partials/footer/center.html @@ -0,0 +1,2 @@ +© {{ now.Year }} {{ .Site.Params.copyright }} +

For website terms of use, trademark policy and other project policies please see lfprojects.org/policies.

From e6224c01eca6a97da5a1d7b028e21889b8c292db Mon Sep 17 00:00:00 2001 From: Andrei Kvapil Date: Thu, 8 Oct 2026 23:48:52 +0200 Subject: [PATCH 16/21] fix(site): make code blocks readable in the dark theme Chroma wrote its light palette inline, so the dark theme put light text on a light block. Highlighting now emits classes, and Docsy's light and dark chroma styles switch with the theme. Assisted-by: LLM Signed-off-by: Andrei Kvapil --- site/assets/scss/_styles_project.scss | 3 +++ site/hugo.yaml | 4 +++- 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/site/assets/scss/_styles_project.scss b/site/assets/scss/_styles_project.scss index d5ec6bf5..239a91c6 100644 --- a/site/assets/scss/_styles_project.scss +++ b/site/assets/scss/_styles_project.scss @@ -108,3 +108,6 @@ display: block; } } + +// Code blocks follow the light and dark theme (needs highlight.noClasses: false). +@import "td/code-dark"; diff --git a/site/hugo.yaml b/site/hugo.yaml index ff0c8f1a..a9eb44b1 100644 --- a/site/hugo.yaml +++ b/site/hugo.yaml @@ -34,7 +34,9 @@ markup: renderer: unsafe: true highlight: - style: friendly + # Classes instead of inline colours, so the stylesheet can switch the + # palette with the light and dark theme. + noClasses: false menu: main: From cf42e5d3ca6c8e14ada0239e48f66752c09b2b28 Mon Sep 17 00:00:00 2001 From: Andrei Kvapil Date: Thu, 8 Oct 2026 23:53:42 +0200 Subject: [PATCH 17/21] docs(site): leave the stale known-issues page off the site Almost every entry on it was fixed within hours of it being written, and what remains is internal validation tracking, not something an operator can run into. Assisted-by: LLM Signed-off-by: Andrei Kvapil --- site/content/en/docs/known-issues.md | 329 --------------------------- 1 file changed, 329 deletions(-) delete mode 100644 site/content/en/docs/known-issues.md diff --git a/site/content/en/docs/known-issues.md b/site/content/en/docs/known-issues.md deleted file mode 100644 index cc6049ee..00000000 --- a/site/content/en/docs/known-issues.md +++ /dev/null @@ -1,329 +0,0 @@ ---- -title: "Known issues" -weight: 60 -description: "Open problems and their workarounds." ---- - - - -This document tracks outstanding bugs and observations surfaced during the scenario/e2e validation session. Each entry references the scenario or test where it was first observed, plus the closing commit if any. - -Severity legend: - -- **P0** — data corruption, split-brain, lost volumes, or cluster-wide unavailability -- **P1** — feature unusable, retry/cancel paths missing, regression vs upstream LINSTOR -- **P2** — cosmetic, log noise, doc/RBAC drift, ergonomic gaps - -## Bug 32: force-strip ZVOL cleanup race - -**Status**: open (observation, not a defect per se) -**Severity**: P2 -**Scenario reference**: tests/scenarios/05-finalizers.md §5.34 -**Surfaced by**: scenario -**Reproduction steps**: - -1. Create a Resource with a ZFS-backed VolumeDefinition on a single node. -2. Force-delete the Resource CR (`kubectl delete --grace-period=0 --force`) while the satellite is still mid-`drbdsetup down`. -3. Observe the underlying ZVOL on the host (`zfs list -t volume`). - -**Expected behaviour**: ZVOL is destroyed in lock-step with the DRBD resource teardown; no stray dataset survives the CR removal. - -**Actual behaviour**: A short window exists where the CR is gone but the ZVOL still resides on disk because the satellite finalizer was bypassed. The next reconcile or operator restart eventually GCs it, but for a few seconds the storage pool reports phantom capacity. - -**Recommended fix**: No code change required. Document the window and rely on Bug 33 (DRBD orphan sweeper) plus Bug 43 (storage orphan sweeper) to handle the residual cleanup deterministically. - -**Related commits / tests**: see Bug 33 (DRBD sweeper) and Bug 43 (storage sweeper). - -## Bug 33: orphan kernel DRBD-Diskless after force-strip - -**Status**: closed (by Bug 5.34 sweeper) -**Severity**: P1 -**Scenario reference**: tests/scenarios/05-finalizers.md §5.34 -**Surfaced by**: scenario -**Reproduction steps**: - -1. Create a 3-replica Resource (2 disk + 1 diskless tiebreaker). -2. Force-delete the tiebreaker Resource CR before the satellite finalizer drains. -3. Inspect `drbdsetup status` on the former diskless node. - -**Expected behaviour**: Kernel DRBD device on the diskless node is fully removed; `drbdsetup status` no longer lists it. - -**Actual behaviour**: Kernel resource remained in `Diskless` state with no corresponding CR, preventing subsequent recreation on the same node (port/minor collision). - -**Recommended fix**: Periodic sweeper on the satellite enumerates kernel DRBD resources and reconciles against the local CR cache; any kernel resource without a matching CR is `drbdsetup down`-ed. - -**Related commits / tests**: 5.34 sweeper landed in the satellite controller; e2e covers via scenario 05 force-strip path. - -## Bug 34: migrate-disk endpoint missing - -**Status**: closed (by 05b8709); Option B follow-up open -**Severity**: P1 -**Scenario reference**: tests/scenarios/07-toggle-disk.md §7.4 -**Surfaced by**: e2e test -**Reproduction steps**: - -1. Create a 2-replica Resource on nodes A and B. -2. Issue `linstor resource-definition migrate-disk `. - -**Expected behaviour**: REST handler accepts the request, schedules a synchronous disk migration (toggle-disk on, then off on source) without losing redundancy. - -**Actual behaviour** (pre-fix): Endpoint returned 404; the upstream LINSTOR client could not orchestrate disk migration through the apiserver. - -**Recommended fix (landed)**: REST handler `migrate-disk` wired in 05b8709 to a two-step toggle-disk pipeline (add on target → wait UpToDate → remove on source). - -**Option B follow-up (open)**: Wrap the two-step sequence in a single reconciler-driven state machine so an apiserver restart mid-migration does not strand the user with an extra diskful replica. Track at next sprint. - -**Related commits / tests**: 05b8709; scenario 07 §7.4 covers the happy path. - -## Bug 35: placer FreeCapacity gate missing - -**Status**: open -**Severity**: P0 -**Scenario reference**: tests/scenarios/08-placer.md §8.2 -**Surfaced by**: scenario -**Reproduction steps**: - -1. Create two storage pools with vastly different free capacity (e.g. 100 GiB and 1 GiB free). -2. Create a Resource of size 10 GiB without an explicit node selector. - -**Expected behaviour**: Placer rejects the small pool because the requested size exceeds its FreeCapacity and chooses the larger pool (or fails fast with a clear error). - -**Actual behaviour**: Placer ignores FreeCapacity entirely and may select the smaller pool, leading to `ENOSPC` at ZVOL/LV create time and a stuck Resource in `Creating`. - -**Recommended fix**: Add a hard FreeCapacity gate inside the candidate-pool filter before scoring. Treat pools with `free < requested + reservedHeadroom` as ineligible. Add a placer unit test with two pools at different free sizes. - -**Related commits / tests**: none yet; blocker for placer 1.0. - -## Bug 36: handleVDUpdate wholesale-replace - -**Status**: open -**Severity**: P1 -**Scenario reference**: tests/scenarios/09-vd-update.md §9.1 -**Surfaced by**: agent -**Reproduction steps**: - -1. Create a VolumeDefinition with `props: {a: 1, b: 2}`. -2. PATCH the VD with `override_props: {a: 9}` (intent: change only `a`). - -**Expected behaviour**: Final props are `{a: 9, b: 2}` — only the keys named in `override_props` are touched. - -**Actual behaviour**: `handleVDUpdate` rewrites the entire props map from the request body, so `b` is silently dropped. - -**Recommended fix**: Replace the wholesale assignment with a merge: apply `override_props` keys on top of the existing map, then remove keys named in `delete_props`. Mirrors the upstream LINSTOR controller semantics and is the precondition for Bug 37. - -**Related commits / tests**: none; depends on Bug 37. - -## Bug 37: VolumeDefinitionModify override_props/delete_props no-op - -**Status**: open -**Severity**: P1 -**Scenario reference**: tests/scenarios/09-vd-update.md §9.2 -**Surfaced by**: e2e test -**Reproduction steps**: - -1. Apply `VolumeDefinitionModify` with `override_props: {DrbdOptions/Net/protocol: C}`. -2. GET the VD and inspect `props`. - -**Expected behaviour**: `DrbdOptions/Net/protocol` is set to `C`; `delete_props` removes named keys. - -**Actual behaviour**: Both fields are accepted by the REST schema but ignored by the handler — the props map is unchanged. - -**Recommended fix**: Wire `override_props` and `delete_props` through the same merge path introduced by Bug 36's fix. Add e2e assertions that round-trip both add and remove. - -**Related commits / tests**: depends on Bug 36; scenario 09 §9.2 has the assertion stub. - -## Bug 38: VD shrink accepted without STATE_INFO warning - -**Status**: open -**Severity**: P2 (cosmetic) -**Scenario reference**: tests/scenarios/09-vd-update.md §9.3 -**Surfaced by**: scenario -**Reproduction steps**: - -1. Create a VD of size 10 GiB. -2. PATCH the VD to size 5 GiB. - -**Expected behaviour**: Either the request is rejected, or it is accepted with a `STATE_INFO` / warning in the response indicating that shrinking is unsafe and won't reclaim space on the backing pool until the FS is also shrunk. - -**Actual behaviour**: Request is accepted silently; no warning surfaces to the operator. Backing volume keeps its original allocation. - -**Recommended fix**: Append a `STATE_INFO` entry to the API response when `newSize < currentSize`. No semantic change to storage; purely advisory. - -**Related commits / tests**: none. - -## Bug 39: toggle-disk retry counter missing - -**Status**: open -**Severity**: P1 -**Scenario reference**: tests/scenarios/07-toggle-disk.md §7.6 -**Surfaced by**: agent -**Reproduction steps**: - -1. Initiate `toggle-disk` on a Resource where the target node's storage pool is briefly unavailable. -2. Observe reconciler logs. - -**Expected behaviour**: Reconciler retries with bounded backoff and a visible retry counter on the Resource status; after N attempts it surfaces a terminal error condition. - -**Actual behaviour**: Reconciler retries forever with no visibility; if the failure is permanent, the Resource is stuck "in flight" indefinitely. - -**Recommended fix**: Add `status.toggleDisk.attempts` and `status.toggleDisk.lastError`. Cap at e.g. 10 attempts before transitioning to `Failed` with a clear condition. Couple with Bug 40. - -**Related commits / tests**: none; precondition for the migrate-disk Option B state machine. - -## Bug 40: toggle-disk cancel state machine missing - -**Status**: open -**Severity**: P1 -**Scenario reference**: tests/scenarios/07-toggle-disk.md §7.7 -**Surfaced by**: agent -**Reproduction steps**: - -1. Start a long-running `toggle-disk` (e.g. large initial sync). -2. Issue a cancel/abort via the REST API. - -**Expected behaviour**: Reconciler observes the cancel intent, rolls back the partial change (removes the half-added diskful, or re-adds the half-removed one) and returns the Resource to its pre-toggle state. - -**Actual behaviour**: No cancel endpoint exists; the only escape is to delete the Resource entirely. - -**Recommended fix**: Model toggle-disk as an explicit state machine (`Pending → Syncing → Promoting → Done | Cancelling → Rolled-back`). Add a cancel verb in the REST surface and have the reconciler honour it idempotently. - -**Related commits / tests**: none; pairs with Bug 39. - -## Bug 41: RBAC nodes get/list/watch missing in stand yaml - -**Status**: closed (13722e1) -**Severity**: P2 -**Scenario reference**: tests/scenarios/01-bootstrap.md §1.3 -**Surfaced by**: e2e test (stand bring-up) -**Reproduction steps**: - -1. Deploy blockstor on the dev stand with the previous controller-manifests yaml. -2. Watch controller logs at startup. - -**Expected behaviour**: Controller starts and lists Node objects to seed its topology cache. - -**Actual behaviour** (pre-fix): `nodes "" is forbidden: User "system:serviceaccount:..." cannot list resource "nodes" in API group "" at the cluster scope`; controller crashloops. - -**Recommended fix (landed)**: Add `nodes` verbs `get/list/watch` to the ClusterRole shipped with the stand yaml. - -**Related commits / tests**: 13722e1. - -## Bug 42: piraeus pod-CIDR drift on e2e-iptables - -**Status**: open (investigation pending) -**Severity**: P1 -**Scenario reference**: tests/scenarios/10-e2e-iptables.md §10.1 -**Surfaced by**: e2e test -**Reproduction steps**: - -1. Bring up the e2e-iptables Talos+QEMU stand via `make iter`. -2. Deploy piraeus-operator with the default values. -3. Wait for the satellite DaemonSet pods to schedule and the LINSTOR controller to register them. - -**Expected behaviour**: Satellite pod IPs come from the cluster's pod CIDR and are routable to/from the LINSTOR controller; resources reach `UpToDate` without DRBD net errors. - -**Actual behaviour**: Some satellite pods receive addresses outside the expected pod CIDR (suspected CNI bridge / kube-proxy iptables drift on the iptables-mode stand), DRBD peer connections flap, resources oscillate between `Connecting` and `Connected`. - -**Recommended fix**: Investigation pending. First steps: - -- Confirm whether the drift is at the CNI layer (bridge IPAM) or kube-proxy (SNAT/MASQUERADE on the iptables stand). -- Capture `ip a`, `iptables-save`, and `kubectl get pods -o wide` on a failing run. -- Compare against the ipvs-mode stand where the issue does not reproduce. - -**Related commits / tests**: scenario 10 §10.1 reproducer; no fix yet. - -## Bug 49: satellite ignores runtime NetInterface changes (deferred from scenario 3.10) - -**Status**: open (deferred — design gap, not a defect) -**Severity**: P2 -**Scenario reference**: tests/scenarios/03-networking.md §3.10 -**Surfaced by**: scenario audit (UG9 §"Managing network interface cards", lines 2167-2169) -**Reproduction steps**: - -1. `kubectl edit node.blockstor.io/` and re-order the entries under `spec.netInterfaces` (or change the first entry's `address`). -2. Observe the running satellite Pod on that worker — it does not re-dial anything, does not re-render its DRBD `on ` block, and continues talking to the kube-apiserver via the same Service VIP. -3. Restart the satellite Pod (`kubectl delete pod -l app=blockstor-satellite,kubernetes.io/hostname=`) — only on restart does the new `Spec.NetInterfaces[0].Address` flow into `LocalAddress` (via `$POD_IP`) and from there into the next reconcile-rendered .res file. - -**Expected behaviour (upstream LINSTOR contract)**: `linstor node interface modify ... --active` flips `StltConn/0/Active=true` and the satellite re-dials the controller via the freshly-active NIC without a process restart. - -**Actual behaviour (blockstor)**: No re-dial happens, and there is no field to flip. Two architectural reasons: - -1. Phase 10.6 retired the satellite→controller gRPC wire entirely. The satellite now talks to the kube-apiserver via the standard in-cluster config (`ctrl.GetConfig()` → `kubernetes.default.svc` Service VIP). There is no LINSTOR-style "controller endpoint" left to re-target. -2. `api/v1alpha1.NodeNetInterface` has no `Active` field. The `IsActive` flag on the REST wire is synthesised at conversion time (`i == 0` in `pkg/store/k8s/nodes.go`) — pure presentation, no behaviour. The only satellite code that reads peer NetInterfaces at runtime is `pkg/satellite/controllers/snapshot_fetcher.go` (peer-to-peer snapshot transport), and it picks by `Name == "default"` or first non-empty `Address`, not by an Active flag. - -**Recommended fix**: Deferred. Implementing Outcome A (live NIC switch) would require: - -- Adding an `Active` field to `NodeNetInterface` and a controller-runtime watch in the satellite that triggers reconciliation when `Spec.NetInterfaces[].Active` changes. -- A redirect plane that the satellite actually re-routes through — but the satellite's only outbound wire is the kube-apiserver REST/watch, which is owned by client-go and routes via the cluster Service VIP rather than this node's NIC selection. -- For the satellite→satellite snapshot stream the change would be marginal (peer-side already picks Address dynamically per Fetch call). - -Net: implementing the upstream contract requires either resurrecting a custom controller-bind wire (rejected by the Phase 10.6 design) or accepting that "active interface" is now a pure presentation field that operators must not rely on for connection re-targeting. - -The spec is pinned by `TestSatelliteFlagsLackControllerBindAddress` and `TestNodeCRDHasNoActiveField` in `pkg/satellite/stream/redial_spec_test.go`. If a future change implements Outcome A, both tests will fail and force the reviewer to replace them with a positive re-dial assertion. - -**Related commits / tests**: `pkg/satellite/stream/redial_spec_test.go`; tests/scenarios/03-networking.md §3.10. - -## Bug 43: mass-churn ZVOL leak on force-stripped finalizers - -**Status**: closed (by storage-orphan sweeper) -**Severity**: P0 -**Scenario reference**: tests/e2e/lc-rd-delete-churn.sh (scenario 5.4 extended — 10 RD rapid create/delete on one node) -**Surfaced by**: e2e test - -**Reproduction steps**: - -1. Run `lc-rd-delete-churn.sh` with `ITERS=10` against a 2-node stand. -2. Inject a satellite stall (e.g. SIGSTOP on the satellite pod) for the iteration's DRBD-down phase so the REST controller times out the per-iter delete and force-strips the Resource CRD finalizer. -3. After the loop, `zfs list -t volume` on each node. - -**Expected behaviour**: All 10 ZVOLs are destroyed in lock-step with the Resource CRD deletion; the final `zfs list -t volume` shows no `churn-N_NNNNN` datasets. - -**Actual behaviour** (pre-fix): 9 of 10 ZVOLs leak. The REST controller stamps `DeletionTimestamp` on the per-node Resource CRD; the satellite finalizer is mid-`drbdsetup down` so doesn't release the finalizer in time; the controller force-strips the finalizer; the satellite's `handleDelete` (which would have called `provider.DeleteVolume`) never runs; the ZVOL survives indefinitely. Bug 32 documents the short-window variant; Bug 43 is the deterministic mass-churn variant where the satellite genuinely cannot keep up and force-strip becomes the steady-state path. - -**Recommended fix (landed)**: New `StorageOrphanSweeperRunnable` in `pkg/satellite/controllers/storage_sweeper.go` periodically (every 5 minutes, same cadence as the DRBD orphan sweeper for Bug 33) walks every registered `storage.Provider` that implements the new `storage.VolumeLister` capability, diffs the on-disk volume list against the Resource CRDs scheduled to this node, and calls `provider.DeleteVolume` on each volume whose owning CRD has fully vanished (force-strip aftermath). - -Safety belts on the new path: - -- Wildcard-owned protection: a Resource CRD whose RD has been cascade-deleted out of order still counts as "owned" (volume number wildcard) so the sweeper never races `handleDelete`. -- Per-cycle delete bound (`StorageSweeperMaxDeletePerCycle = 3`) so a pathological upstream producer of orphans surfaces in logs without being silently cleaned in one tick. -- Storage-resource prefix allowlist (`pvc-`, `bs-`) so operator-created datasets that happen to share the pool are immune to the sweep. -- Per-Node opt-out annotation `blockstor.io/skip-orphan-storage-sweeper` for forensic-mode operators. -- `ListVolumeNames` skips ZFS deferred-delete markers (`__DELETED__`) and LVM snapshot LVs via the lv_attr column so the sweeper never confuses cleanup artefacts for orphans. - -Idempotency of `provider.DeleteVolume` was already guaranteed by Bug 33's contract; the sweeper relies on that — racing the satellite's own `handleDelete` is harmless because the second call returns nil. - -**Related commits / tests**: `pkg/satellite/controllers/storage_sweeper.go` + `pkg/satellite/controllers/storage_sweeper_test.go` (7 unit tests covering owned-leave-alone, orphan-reap, foreign-prefix immunity, per-node scope, rate-limit, skip-annotation, mid-delete-race protection, missing-RD wildcard protection); `pkg/storage/storage.go` adds the `VolumeLister` optional interface + `VolumeRef`; ZFS / LVM-thin / LVM-thick providers implement it. E2e validation via `lc-rd-delete-churn.sh` (passes 10 iters without ZVOL leak). - -## Bug 50: concurrent rapid late `vd c` wedges or drops the second volume - -**Status**: closed (#164 converge + #168 resize-deadlock guard; v0.1.14) (campaign tracking id: BUG-048) -**Severity**: P1 (availability; NOT data-loss, NOT a node-reboot deadlock; recoverable) -**Scenario reference**: tests/e2e/cli-matrix/multi-volume-late-vd-create.sh -**Surfaced by**: release-gate validation campaign -**Reproduction steps**: - -1. Create a multi-replica ResourceDefinition and let it converge (at least one VolumeDefinition, all replicas UpToDate). -2. Issue two `linstor volume-definition create ` calls back-to-back in rapid succession (concurrently, or scripted with no wait between them). - -**Expected behaviour**: Both volume-definitions are created; each new volume is laid down on every replica and reaches `UpToDate` (with one replica electing SyncSource), exactly as a sequential add would. - -**Actual behaviour**: Intermittently one of two faults occurs — (a) the second `vd c` returns SUCCESS but the VolumeDefinition is silently dropped (only one VD ends up persisted), or (b) the second volume is created but wedges `Inconsistent` with no SyncSource elected, so it never reaches `UpToDate`. In case (a) both calls report success. - -**Scope / reachability**: NOT reachable through the production CSI path. linstor-csi maps one PVC to one ResourceDefinition with exactly one VolumeDefinition, created in a single request; autoplace and `ControllerPublishVolume` add no VolumeDefinitions, and resize grows the existing volume in place. The fault is reachable only when an operator (or a script) issues concurrent / rapid back-to-back manual `vd c` calls against an existing multi-replica RD. No data is lost or corrupted (md5 held across all validation runs) and no node reboot is required. - -**Recovery**: Delete the wedged or missing volume-definition (`linstor vd d `) and re-add it sequentially — issue one `vd c`, wait for it to reach `UpToDate`, then issue the next. Always add late volume-definitions one at a time. - -**Recommended fix**: Serialise concurrent VolumeDefinition creation on a given ResourceDefinition (optimistic-concurrency precondition on the RD volume-minor allocation plus the auto-numbered VD create) so the second add observes the first, and seed the new volume's GI / current-UUID before peers attach so a deterministic SyncSource is elected. A first attempt (PR #164) reduced the silent-drop and fixed a day0 split-brain leg but regressed the CSI-reachable single-late-VD path, so it was not merged; a corrected fix that does not touch the single-VD path is tracked. - -**Related commits / tests**: cli-matrix cell `tests/e2e/cli-matrix/multi-volume-late-vd-create.sh` honestly catches this (as of #162); PR #164 (deferred). No closing commit yet. - -## Recommended next-fix order - -1. **Bug 35** (P0, placer FreeCapacity) — only P0; blocks placer 1.0 and risks ENOSPC at create time. -2. **Bug 42** (P1, piraeus pod-CIDR drift) — blocks the iptables-mode e2e lane. -3. **Bug 36 + 37** (P1, VD props merge) — fix together; 37 depends on 36's merge plumbing. -4. **Bug 39 + 40** (P1, toggle-disk retry/cancel) — fix together; together they unlock Bug 34's Option B state machine. -5. **Bug 50** (FIXED in v0.1.14, #164+#168) — serialise VolumeDefinition create on the RD and seed GI before peer attach; operator-only path, recoverable, not data-loss. Closed by #164 (converge) + #168 (resize-deadlock guard). -6. **Bug 34 Option B** (P1 follow-up) — wrap migrate-disk in the new state machine once 39/40 land. -7. **Bug 38** (P2, cosmetic STATE_INFO on shrink) — pure UX, do last. -8. **Bug 32** (P2, observation) — document only; no code fix needed. From f81d69566a6672e6a415cbdcf773aa3a75a78dd7 Mon Sep 17 00:00:00 2001 From: Andrei Kvapil Date: Fri, 9 Oct 2026 00:07:37 +0200 Subject: [PATCH 18/21] docs(site): rewrite the documentation for operators The pages copied from docs/ were written for people working on the code: they explained where things live in the tree and drove everything through the upstream linstor client. The documentation section is now organised around what an operator does: overview, installation, CSI, resources with examples, the blockstor CLI, the layer stack, migration from LINSTOR and the differences from LINSTOR. The cover's Get started button points at the installation page. Assisted-by: LLM Signed-off-by: Andrei Kvapil --- site/content/en/_index.md | 2 +- site/content/en/docs/_index.md | 11 +- site/content/en/docs/cli.md | 154 +++++++++++ site/content/en/docs/compatibility.md | 47 ++++ site/content/en/docs/csi.md | 101 ++++++++ site/content/en/docs/getting-started.md | 23 -- site/content/en/docs/installation.md | 141 ++++++++++ site/content/en/docs/layer-stack.md | 148 ++++------- site/content/en/docs/linstor-migration.md | 130 ---------- site/content/en/docs/migration.md | 103 ++++++++ site/content/en/docs/overview.md | 57 ++++ site/content/en/docs/resources.md | 201 ++++++++++++++ site/content/en/docs/usage.md | 302 ---------------------- 13 files changed, 868 insertions(+), 552 deletions(-) create mode 100644 site/content/en/docs/cli.md create mode 100644 site/content/en/docs/compatibility.md create mode 100644 site/content/en/docs/csi.md delete mode 100644 site/content/en/docs/getting-started.md create mode 100644 site/content/en/docs/installation.md delete mode 100644 site/content/en/docs/linstor-migration.md create mode 100644 site/content/en/docs/migration.md create mode 100644 site/content/en/docs/overview.md create mode 100644 site/content/en/docs/resources.md delete mode 100644 site/content/en/docs/usage.md diff --git a/site/content/en/_index.md b/site/content/en/_index.md index cadb539f..c9d95297 100644 --- a/site/content/en/_index.md +++ b/site/content/en/_index.md @@ -7,7 +7,7 @@ title: Blockstor

Blockstor

A Kubernetes control plane for LVM and ZFS storage with DRBD replication.

-Get started +Get started GitHub
diff --git a/site/content/en/docs/_index.md b/site/content/en/docs/_index.md index c1e450b3..332d0ef1 100644 --- a/site/content/en/docs/_index.md +++ b/site/content/en/docs/_index.md @@ -8,6 +8,13 @@ menu: description: "Install, operate and understand blockstor." --- -Blockstor is a Kubernetes control plane for LVM and ZFS storage with DRBD replication. It speaks a LINSTOR-compatible REST API, so `linstor-csi`, `piraeus-operator`, `ha-controller`, `golinstor` and the `linstor` CLI keep working unchanged. +Blockstor is a Kubernetes control plane for LVM and ZFS storage with DRBD replication. It speaks a LINSTOR-compatible REST API, so `linstor-csi`, `piraeus-operator`, `ha-controller` and `golinstor` keep working unchanged, and it ships its own `blockstor` command-line tool. -Start with [Getting started](getting-started/), then see [Usage](usage/) for the full walkthrough. +Where to start: + +- New to blockstor: read the [Overview](overview/), then [Architecture](architecture/). +- Setting up a cluster: [Installation](installation/), then [Provisioning volumes with CSI](csi/). +- Day-to-day work: [Using the CLI](cli/) and [Resources](resources/). +- Coming from LINSTOR: [Migrating from LINSTOR](migration/) and [Compatibility with LINSTOR](compatibility/). + +If you want to work on blockstor itself, see the [Developer guide](/developer-guide/). diff --git a/site/content/en/docs/cli.md b/site/content/en/docs/cli.md new file mode 100644 index 00000000..8a35a901 --- /dev/null +++ b/site/content/en/docs/cli.md @@ -0,0 +1,154 @@ +--- +title: "Using the CLI" +linkTitle: "Using the CLI" +weight: 60 +description: "Manage nodes, pools, volumes and snapshots with the blockstor command." +--- + +`blockstor` is the command-line tool for blockstor. It reads your kubeconfig and works on the blockstor custom resources directly, so it needs no port-forward and no API certificate: if `kubectl` can reach the cluster, so can `blockstor`. + +The commands follow the `linstor` client's noun-verb grammar, with the same short aliases, so runbooks written for LINSTOR mostly carry over by changing the command name. Tools that speak the LINSTOR REST API, such as `linstor-csi`, keep working against the blockstor API server. The examples on this page use `blockstor`. + +## Install + +```sh +go install github.com/cozystack/blockstor/cmd/blockstor@v +``` + +The tool uses the same kubeconfig as `kubectl`: the `KUBECONFIG` variable, then `~/.kube/config`, or the service account when it runs inside a pod. + +## Getting help + +```sh +blockstor --help # every object and action +blockstor resource create --help # usage of one command +``` + +Every object and action has a short alias: `blockstor sp l` is `blockstor storage-pool list`, `blockstor rd c` is `blockstor resource-definition create`, `blockstor r l` is `blockstor resource list`. + +Output options that work on list commands: + +- `-m` or `--machine-readable` prints JSON. +- `--color=auto|always|never` controls colours. They are on only for a terminal, and `NO_COLOR` turns them off. + +## Nodes and storage pools + +```sh +blockstor node create worker-1 10.0.0.11 +blockstor node list + +blockstor physical-storage list +blockstor physical-storage create-device-pool zfs worker-1 /dev/sdb --pool-name data +blockstor storage-pool create lvmthin worker-2 data my_vg/my_thinpool +blockstor storage-pool list +``` + +Take a node out of service. `evacuate` moves its replicas to other nodes; `restore` brings it back: + +```sh +blockstor node evacuate worker-2 +blockstor node restore worker-2 +``` + +## Resource groups + +A resource group holds the policy, and every volume spawned from it gets the same settings: + +```sh +blockstor resource-group create replicated --place-count 2 --storage-pool data +blockstor volume-group create replicated +blockstor resource-group list +``` + +Change the policy later with `blockstor resource-group modify`, for example `--place-count 3`, then apply it to the volumes that already exist with `blockstor resource-group adjust`. + +## Volumes + +Spawn a volume from a group in one step. This creates the definition, its volume and the replicas: + +```sh +blockstor resource-group spawn replicated my-volume 10G +``` + +Or step by step, without a group policy: + +```sh +blockstor resource-definition create my-volume +blockstor volume-definition create my-volume 10G +blockstor resource create my-volume --auto-place 2 --storage-pool data +``` + +To put replicas on named nodes instead of letting blockstor choose: + +```sh +blockstor resource create worker-1 worker-2 my-volume --storage-pool data +``` + +Inspect the state: + +```sh +blockstor resource-definition list +blockstor resource list # replicas, nodes and DRBD state +blockstor resource list --faulty # only replicas that are not healthy +blockstor volume list # sizes and device paths +``` + +A healthy replicated volume shows `UpToDate` on every replica with data, and a `TieBreaker` when there is an even number of them. + +Grow a volume online. Shrinking is refused unless you add `--force`, because it cuts off whatever data lives at the end of the device: + +```sh +blockstor volume-definition set-size my-volume 0 20G +``` + +Turn a diskless replica into one with data, or the other way round: + +```sh +blockstor resource toggle-disk worker-3 my-volume --storage-pool data +blockstor resource toggle-disk worker-3 my-volume --diskless +``` + +Delete a volume with all its replicas: + +```sh +blockstor resource-definition delete my-volume +``` + +## Snapshots + +```sh +blockstor snapshot create my-volume before-upgrade +blockstor snapshot list +``` + +Restore a snapshot as a new volume. This creates the new definition, fills its volumes from the snapshot and places the replicas. The original volume is not touched: + +```sh +blockstor snapshot resource restore \ + --from-resource my-volume --from-snapshot before-upgrade --to-resource my-volume-restored +``` + +`blockstor snapshot resource-definition restore` is another spelling of the same command. To copy a live volume without taking a snapshot yourself, use `blockstor resource-definition clone my-volume my-volume-copy`. + +In-place rollback is not available on purpose: it destroys every snapshot newer than the one you roll back to. Restore into a new volume instead. + +```sh +blockstor snapshot delete my-volume before-upgrade +``` + +## Properties and DRBD options + +Every object has `set-property`, `list-properties` and `delete-property`, and groups, definitions and the controller have `drbd-options`. Settings resolve from the controller to the group to the definition to the replica, and the narrowest one wins. See [Resources](../resources/#how-settings-are-inherited). + +```sh +blockstor resource-group set-property replicated DrbdOptions/Resource/on-no-quorum suspend-io +blockstor resource-definition list-properties my-volume +``` + +## Encryption + +```sh +blockstor encryption create-passphrase '' +``` + +See [Layer stack](../layer-stack/#encryption) before you use it. diff --git a/site/content/en/docs/compatibility.md b/site/content/en/docs/compatibility.md new file mode 100644 index 00000000..c9f2de70 --- /dev/null +++ b/site/content/en/docs/compatibility.md @@ -0,0 +1,47 @@ +--- +title: "Compatibility with LINSTOR" +linkTitle: "Compatibility with LINSTOR" +weight: 80 +description: "Where blockstor behaves differently from LINSTOR, and what is not there." +--- + +Blockstor implements the LINSTOR REST API, so `linstor-csi`, `piraeus-operator`, `ha-controller`, `golinstor` and the `linstor` client work against it. This page lists the differences an operator can notice. The full list, kept for the project's parity tests, is in [`docs/cli-parity-known-deltas.md`](https://github.com/cozystack/blockstor/blob/main/docs/cli-parity-known-deltas.md). + +## Not implemented + +| Feature | What happens | +|---|---| +| Backups, backup shipping, remotes such as S3 | The API answers `501 Not Implemented`. Use snapshots and restore inside the cluster. | +| Schedules | The API answers `501 Not Implemented`. | +| `advise` | Not available. | +| In-place snapshot rollback | Refused on purpose, because it destroys every newer snapshot. Restore the snapshot into a new volume instead. | +| Per-connection DRBD peer options (`resource-connection drbd-peer-options`) | Not available. | +| External DRBD metadata (`StorPoolNameDrbdMeta`) | Not supported yet. | +| Other providers: SPDK, NVMe-oF, OpenFlex, Exos | Not supported. | + +## Behaves differently + +| Area | LINSTOR | Blockstor | +|---|---|---| +| Active network interface | `linstor node interface modify --active` switches the satellite's connection to another interface. | Has no effect. The first interface of a node is always the active one; to make another interface active, put it first. A change of address reaches DRBD after the node's satellite pod restarts. | +| Default action on lost quorum | `on-no-quorum io-error` | `on-no-quorum suspend-io`: I/O blocks until quorum returns and then resumes, so a replica that was cut off does not get stuck after the network heals. | +| Shrinking a volume | Allowed for stacks that support it. | Refused unless you pass `--force`. | +| DRBD ports and minors | Ports from 7000, minors from 1000. | Ports from 20000 to 20999, minors from 20000. Blockstor can run on the same nodes as LINSTOR without collisions. Change the port range per node with the `DrbdOptions/TcpPortRange` property. | +| Spawning more replicas than there are nodes | Fails and places nothing. | Places as many replicas as it can, reports the shortfall, and adds the rest when more nodes join. | +| Turning a tiebreaker into a replica with data | Can skip the initial sync. | Runs a full sync, to avoid a DRBD split where both sides refuse to connect. | +| Removing a replica below quorum | Removes the quorum options from the definition and prints a note. | Sets `quorum off` explicitly, with the same effect on DRBD. | +| DRBD option values | Validated against the DRBD schema. | Not validated: a wrong value is stored and fails later on the node. Check values before you set them. | +| Autoplacer weights | `MaxFreeSpace` is 1, the other weights 0. | All four weights default to 1. | +| `controller version` | Reports a git commit hash. | Reports `git=blockstor`. Do not parse a commit hash from it. | +| `node info` | Shows OS, CPU and memory. | These fields are empty. | + +## Using the `linstor` client + +The `linstor` client works against blockstor's API server. Port-forward the plain HTTP port, which is reachable only from inside the pod, and point the client at it: + +```sh +kubectl -n blockstor-system port-forward deploy/blockstor-apiserver 3370:3370 +linstor --controllers http://localhost:3370 node list +``` + +For everyday work the [`blockstor` CLI](../cli/) is simpler: it needs no port-forward and no certificates, and it accepts the same commands. diff --git a/site/content/en/docs/csi.md b/site/content/en/docs/csi.md new file mode 100644 index 00000000..393339c9 --- /dev/null +++ b/site/content/en/docs/csi.md @@ -0,0 +1,101 @@ +--- +title: "Provisioning volumes with CSI" +linkTitle: "CSI and StorageClasses" +weight: 25 +description: "Connect linstor-csi to blockstor and provision PersistentVolumes." +--- + +Kubernetes provisions blockstor volumes through the LINSTOR CSI driver, `linstor-csi`. The driver speaks the LINSTOR REST API, and blockstor serves that API, so you point the driver at the blockstor API server instead of a LINSTOR controller. + +## What to deploy + +Piraeus normally ships a whole LINSTOR distribution. With blockstor you take only the client side: + +| Piraeus component | With blockstor | +|---|---| +| `linstor-controller` | Skip. The blockstor controller and API server replace it. | +| `linstor-satellite` | Skip. Blockstor runs its own satellites. | +| `linstor-csi` (controller and node plugin) | Deploy, pointed at the blockstor API server. | +| HA controller, `drbd-reactor` | Optional. Point them at the API server the same way as CSI. | + +## Point the driver at blockstor + +The API server listens with mutual TLS on port 3371 of the `blockstor-apiserver` Service in `blockstor-system`. The installation created a client certificate for in-cluster clients: the Secret `blockstor-apiserver-client-tls` carries `tls.crt`, `tls.key` and `ca.crt`. Mount it into both the CSI controller and the CSI node plugin, and set the endpoint and certificate paths: + +```yaml +env: + - name: LS_CONTROLLERS + value: "https://blockstor-apiserver.blockstor-system.svc:3371" + - name: LS_USER_CERTIFICATE + value: /etc/linstor/client/tls.crt + - name: LS_USER_KEY + value: /etc/linstor/client/tls.key + - name: LS_ROOT_CA + value: /etc/linstor/client/ca.crt +volumeMounts: + - name: client-tls + mountPath: /etc/linstor/client + readOnly: true +volumes: + - name: client-tls + secret: + secretName: blockstor-apiserver-client-tls +``` + +The node plugin's `--node` argument must match the name of a blockstor `Node`, which is the Kubernetes node name. + +If you install `linstor-csi` with the piraeus operator, set `LinstorCluster.spec.externalController.url` to `https://blockstor-apiserver.blockstor-system.svc:3371`. That turns off the operator's own LINSTOR controller and points the CSI driver at blockstor. Point `spec.apiTLS.certManager` at the `blockstor-api-ca` Issuer, so the operator issues the client certificate for the CSI pods. + +The Secret lives in `blockstor-system`. If the CSI driver runs in another namespace, issue a second client certificate into that namespace from the same `blockstor-api-ca` Issuer. + +cert-manager renews the certificates in place, and the API server picks up a new certificate without a restart. + +## StorageClass + +Use the `linstor.csi.linbit.com` provisioner: + +```yaml +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: blockstor-replicated +provisioner: linstor.csi.linbit.com +parameters: + linstor.csi.linbit.com/storagePool: "data" + linstor.csi.linbit.com/placementCount: "2" + # Optional: take the policy from a resource group you created. + # linstor.csi.linbit.com/resourceGroup: "replicated" +volumeBindingMode: WaitForFirstConsumer +allowVolumeExpansion: true +``` + +- `storagePool` is the pool that holds the replicas with data. +- `placementCount` is the number of replicas with data. Blockstor adds a tiebreaker when that number is even. +- `resourceGroup` uses a [resource group](../resources/#resourcegroup) as the policy. Without it, volumes go into the default group, `DfltRscGrp`. + +`storagePool`, `placementCount` and `resourceGroup` are covered by blockstor's own tests. Other `linstor.csi.linbit.com/*` parameters are passed through to the API unchanged; test the ones you rely on in your cluster first. + +## A first PVC + +```yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: blockstor-test +spec: + accessModes: [ReadWriteOnce] + storageClassName: blockstor-replicated + resources: + requests: + storage: 1Gi +``` + +With `WaitForFirstConsumer`, the PVC binds once a pod that uses it is scheduled. The PersistentVolume's name is also the name of the blockstor resource definition: + +```sh +kubectl get pvc blockstor-test +kubectl get pv +blockstor resource list +``` + +Delete volumes through Kubernetes: deleting the PVC makes the CSI driver delete the definition and its replicas. diff --git a/site/content/en/docs/getting-started.md b/site/content/en/docs/getting-started.md deleted file mode 100644 index 43ba552a..00000000 --- a/site/content/en/docs/getting-started.md +++ /dev/null @@ -1,23 +0,0 @@ ---- -title: "Getting started" -weight: 10 -description: "What blockstor needs and where to begin." ---- - -Blockstor installs onto an existing cluster and is driven with the standard `linstor` client and piraeus `linstor-csi`. The full walkthrough, from installing the control plane to registering nodes and pools and wiring up CSI, is in [Usage](../usage/). - -## Images - -The three images are published to GHCR on every release: - -- `ghcr.io/cozystack/blockstor-controller`: the reconcilers -- `ghcr.io/cozystack/blockstor-apiserver`: the LINSTOR-compatible REST API, over mTLS -- `ghcr.io/cozystack/blockstor-satellite`: the per-node DRBD and storage agent - -## Coming from LINSTOR - -An existing LINSTOR cluster can be moved onto blockstor; see [Migrating from LINSTOR](../linstor-migration/). - -## Contributing - -The repository layout and the local Talos and QEMU dev stand are described in [AGENTS.md](https://github.com/cozystack/blockstor/blob/main/AGENTS.md). diff --git a/site/content/en/docs/installation.md b/site/content/en/docs/installation.md new file mode 100644 index 00000000..8b4fafad --- /dev/null +++ b/site/content/en/docs/installation.md @@ -0,0 +1,141 @@ +--- +title: "Installation" +linkTitle: "Installation" +weight: 20 +description: "Prepare the nodes, deploy blockstor, and check that it is up." +--- + +Blockstor installs onto an existing Kubernetes cluster. You need the storage nodes prepared, cert-manager in the cluster, and the blockstor manifests from the release you want to run. + +## Prerequisites + +On every storage node that will run the satellite: + +- The DRBD 9 kernel module, loaded (`modprobe drbd`). +- For ZFS pools, the ZFS kernel module. On Talos Linux both modules come from the `siderolabs/drbd` and `siderolabs/zfs` system extensions. +- At least one spare block device, or an existing LVM volume group or ZFS pool to back the storage pools. + +The satellite image already carries the userspace tools (`drbd-utils`, `lvm2`, `cryptsetup`, `zfsutils`); only the kernel modules come from the host. + +In the cluster: + +- [cert-manager](https://cert-manager.io/), which issues the certificates for the API's mutual TLS. +- Permission to run privileged pods in the `blockstor-system` namespace. The satellite needs host networking and access to the host's block devices. + +## Get the manifests + +The CRDs and workload manifests are in the blockstor repository. Check out the tag of the release you are installing: + +```sh +git clone https://github.com/cozystack/blockstor.git +cd blockstor +git checkout v +``` + +The workload manifests carry an `__REGISTRY__/:dev` placeholder used by the development stand. Point them at the published images for your version: + +```sh +TAG= +mkdir -p rendered +for f in blockstor-deploy.yaml blockstor-apiserver-tls.yaml blockstor-apiserver-deploy.yaml blockstor-satellite-daemonset.yaml; do + sed -e "s#__REGISTRY__/\(blockstor-[a-z]*\):dev#ghcr.io/cozystack/\1:${TAG}#" "stand/$f" > "rendered/$f" +done +``` + +Each release publishes the images as `MAJOR.MINOR.PATCH` and `MAJOR.MINOR`, and stable releases also as `latest`. + +## Deploy + +Apply the pieces in this order. + +1. The custom resource definitions: + + ```sh + kubectl apply -f config/crd/bases/ + ``` + +2. The namespace, the controller and its RBAC: + + ```sh + kubectl apply -f rendered/blockstor-deploy.yaml + ``` + +3. The certificate chain for the API. It creates a CA, the API server's serving certificate, and a client certificate for in-cluster clients such as CSI: + + ```sh + kubectl apply -f rendered/blockstor-apiserver-tls.yaml + ``` + +4. The API server and the satellites: + + ```sh + kubectl apply -f rendered/blockstor-apiserver-deploy.yaml + kubectl apply -f rendered/blockstor-satellite-daemonset.yaml + ``` + +The satellite DaemonSet runs on every node that is not a control-plane node. Wait for everything to come up: + +```sh +kubectl -n blockstor-system rollout status deploy/blockstor-controller +kubectl -n blockstor-system rollout status deploy/blockstor-apiserver +kubectl -n blockstor-system rollout status daemonset/blockstor-satellite +``` + +## Register nodes and storage pools + +Each storage node is a cluster-scoped `Node` resource named after the Kubernetes node, and each pool on a node is a `StoragePool`. You can create them as YAML (see [Resources](../resources/)) or with the CLI. + +Install the CLI from the release tag. It needs Go and reads your kubeconfig, so it talks to the cluster the same way `kubectl` does: + +```sh +go install github.com/cozystack/blockstor/cmd/blockstor@v +``` + +Register the nodes. The address is the one DRBD uses for replication traffic between nodes: + +```sh +blockstor node create worker-1 10.0.0.11 +blockstor node create worker-2 10.0.0.12 +blockstor node create worker-3 10.0.0.13 +``` + +Create a pool from a raw disk in one step. This prepares the device and registers the pool: + +```sh +blockstor physical-storage create-device-pool zfs worker-1 /dev/sdb --pool-name data +``` + +Or register a volume group or ZFS pool that already exists: + +```sh +blockstor storage-pool create lvmthin worker-1 data my_vg/my_thinpool +``` + +The providers are `lvm`, `lvmthin`, `zfs`, `zfsthin`, `file`, `filethin` and `diskless`. Every node also gets a diskless pool, `DfltDisklessStorPool`, so it can host diskless replicas and tiebreakers. + +## Check the installation + +```sh +blockstor controller version +blockstor node list +blockstor storage-pool list +``` + +Every node should show as online, and every pool should report its free and total capacity once its satellite has seen it. + +Then create a first replicated volume to see the whole path work: + +```sh +blockstor resource-group create mygroup --place-count 2 --storage-pool data +blockstor volume-group create mygroup +blockstor resource-group spawn mygroup test-volume 1G +blockstor resource list +``` + +A healthy result is two `UpToDate` replicas and a `TieBreaker` on a third node. Remove it when you are done: + +```sh +blockstor resource-definition delete test-volume +``` + +Next, connect Kubernetes to it: [Provisioning volumes with CSI](../csi/). diff --git a/site/content/en/docs/layer-stack.md b/site/content/en/docs/layer-stack.md index 426c0302..5d0658d9 100644 --- a/site/content/en/docs/layer-stack.md +++ b/site/content/en/docs/layer-stack.md @@ -1,118 +1,78 @@ --- title: "Layer stack" +linkTitle: "Layer stack" weight: 50 -description: "DRBD, LUKS and STORAGE compositions." +description: "Combine replication, encryption and plain local storage." --- - +Each volume is built from layers. The bottom layer allocates space on the node, and each layer above adds something on top of the one below. The topmost layer is the device the pod uses. -blockstor mirrors LINSTOR's `layer_list` model: a `ResourceDefinition` -declares an ordered chain of layers, each adding capabilities on top -of the layer below. The satellite walks the chain bottom-up when -provisioning and top-down when tearing down. +| Layer | What it does | +|---|---| +| `STORAGE` | Allocates the block device: an LVM or LVM-thin logical volume, a ZFS zvol, or a file. Every stack ends with it. | +| `LUKS` | Encrypts the device below it with `cryptsetup`. | +| `DRBD` | Replicates the device to the other nodes over the network. | -## Layers +## Supported stacks -| Layer | What it does | -|-----------|-----------------------------------------------------------------------| -| `STORAGE` | Allocates the raw block device (LVM-thin LV, ZFS volume, loopfile, …) | -| `LUKS` | `cryptsetup luksFormat` + `luksOpen` over the storage device | -| `DRBD` | Renders `.res` and runs `drbdadm` to replicate over the network | +| Stack | Use it for | +|---|---| +| `DRBD`, `STORAGE` | Replicated volumes. This is the default. | +| `DRBD`, `LUKS`, `STORAGE` | Replicated and encrypted at rest. Each replica encrypts its own copy. | +| `LUKS`, `STORAGE` | Encrypted volume with a single replica and no DRBD. | +| `STORAGE` | Plain local volume with a single replica, for caches and scratch space. | -The first entry in `layerStack` is the topmost layer the consumer Pod -mounts. The last entry is always `STORAGE` (the storage layer is -required — every replica needs a backing device, even diskless ones -where it's the network leg). +These four are the only orders blockstor accepts. Other orders, such as LUKS on top of DRBD, are refused when the definition is created. -## Common compositions +## Choosing the stack -| Stack | Use case | -|-----------------------------|---------------------------------------------------------------------| -| `["DRBD","STORAGE"]` | Default. Replicated PVC, the cozystack production case. | -| `["LUKS","STORAGE"]` | Single-replica encrypted PVC. No DRBD overhead. | -| `["DRBD","LUKS","STORAGE"]` | Encrypted at-rest + replicated. Per-volume cipher. | -| `["STORAGE"]` | Single-replica local mode. Ephemeral cache, scratch, ZFS dataset. | +A definition takes its stack from its own `layerStack`, then from its resource group's `selectFilter.layerStack`. If neither is set, it gets `DRBD`, `STORAGE`. -The default is `["DRBD","STORAGE"]` — when `RD.Spec.LayerStack` is -empty the controller inherits from the parent -`ResourceGroup.Spec.SelectFilter.LayerStack`, and when both are empty -the dispatcher falls through to the satellite's default-DRBD path. +Set it on a group, so every volume spawned from it gets the same stack: -## Setting the stack +```sh +blockstor resource-group create local --place-count 1 --storage-pool data --layer-list storage +blockstor volume-group create local +blockstor resource-group spawn local scratch-1 20G +``` -### Per-RD via REST +Or on one definition, in YAML: -```bash -curl -XPOST http://controller:3370/v1/resource-definitions \ - -H 'Content-Type: application/json' \ - -d '{"resource_definition":{"name":"pvc-1","layer_stack":["LUKS","STORAGE"]}}' +```yaml +apiVersion: blockstor.cozystack.io/v1alpha1 +kind: ResourceDefinition +metadata: + name: scratch-1 +spec: + layerStack: ["STORAGE"] + volumeDefinitions: + - volumeNumber: 0 + sizeKib: 20971520 ``` -### Per-RG via spawn template +Treat the stack as fixed once a volume has replicas. Changing `layerStack` later does not encrypt or decrypt data that is already there. + +## Encryption + +LUKS volumes are opened with the cluster passphrase, which lives in a Secret in `blockstor-system`. Create it once, before the first encrypted volume: -```bash -linstor rg create encrypted-rg -linstor rg sp encrypted-rg DrbdOptions/Encryption/passphrase '<32-byte-secret>' -linstor rg c --place-count 2 encrypted-rg -linstor rg spawn encrypted-rg pvc-2 1G --layer-list LUKS DRBD +```sh +blockstor encryption create-passphrase '' ``` -(blockstor accepts `linstor`'s `--layer-list` flag verbatim because -the REST shape mirrors upstream LINSTOR.) +Keep a copy of the passphrase somewhere safe. Without it, encrypted volumes cannot be opened. Running the command again with the same passphrase is a no-op, and blockstor refuses to replace it with a different one, because that would lock every existing encrypted volume. A definition with `LUKS` in its stack and no passphrase available is refused rather than created unencrypted. -### Per-RD via kubectl +Then create volumes with LUKS in the stack: -```yaml -apiVersion: blockstor.cozystack.io/v1alpha1 -kind: ResourceDefinition -metadata: {name: pvc-encrypted} -spec: - layerStack: ["LUKS", "STORAGE"] - props: - DrbdOptions/Encryption/passphrase: "32-byte-secret" - volumeDefinitions: - - {volumeNumber: 0, sizeKib: 1048576} +```sh +blockstor resource-group create secure --place-count 2 --storage-pool data --layer-list drbd,luks,storage +blockstor volume-group create secure +blockstor resource-group spawn secure secret-volume 10G ``` -## LUKS specifics - -- Passphrase is per-RD via `DrbdOptions/Encryption/passphrase`. The - upstream `linstor rd set-property` key. -- The dispatcher folds the resolver-resolved passphrase onto the wire - as `DesiredResource.Props["LuksPassphrase"]`; the satellite reads it - from there. -- Empty passphrase with `LUKS` in the stack fails the apply rather - than silently producing an unencrypted volume. -- Mapper name: `--luks` → `/dev/mapper/--luks`. Stable - across reconciles so reopen on satellite restart re-uses it. -- Volume grow: the satellite runs `cryptsetup resize` on the mapper - after the storage layer has resized the underlying LV, before DRBD - resizes the replicated device. - -## What blockstor doesn't yet support - -- **Pluggable layer ordering**: the stack must be one of the four - rows in the table above. Arbitrary orderings (e.g. `["LUKS","DRBD","STORAGE"]` - with LUKS-over-DRBD instead of DRBD-over-LUKS) aren't yet validated - or rendered. -- **Per-volume LUKS keys**: every volume on a 2-volume RD currently - uses the same RD-level passphrase. Per-volume keys (with master-key - wrapping in the controller's KV store) is a follow-up. -- **Mid-stack changes**: editing `layerStack` after a Resource is - active doesn't re-encrypt or unwrap existing data. Treat it as - set-once-per-RD until migration support lands. -- **Cluster-passphrase rotation**: the cluster passphrase - (`POST /v1/encryption/passphrase`) wraps RD passphrases in the - controller's KV store; rotating it doesn't re-encrypt LUKS headers. - Rotating the per-RD passphrase requires `cryptsetup luksChangeKey` - which isn't wired yet — operators should drop the RD and recreate. - -## Testing - -- `pkg/api/v1/layer_stack_test.go` — RD → RG → default resolution. -- `pkg/satellite/reconciler_drbd_test.go::TestApplySkipsDRBDWhenLayerStackOmits` - — `["STORAGE"]` produces no `.res` and no drbdadm. -- `pkg/satellite/reconciler_drbd_test.go::TestApplyLayersLUKS` — pin - cryptsetup luksFormat + luksOpen run on first activation. -- `pkg/satellite/reconciler_drbd_test.go::TestApplyLUKSFailsWithoutPassphrase` - — pin the explicit-error path. +Growing an encrypted volume resizes every layer in order: the storage below, then LUKS, then DRBD. + +Limits today: + +- All volumes of a definition share one key. +- Changing the passphrase of existing volumes is not supported. To re-key a volume, create a new one and move the data. diff --git a/site/content/en/docs/linstor-migration.md b/site/content/en/docs/linstor-migration.md deleted file mode 100644 index 418f9c1d..00000000 --- a/site/content/en/docs/linstor-migration.md +++ /dev/null @@ -1,130 +0,0 @@ ---- -title: "Migrating from LINSTOR" -weight: 30 -description: "Move an existing LINSTOR cluster onto blockstor." ---- - - - -`linstor-migrate` converts a LINSTOR k8s-backend database dump into blockstor CRDs. blockstor then **adopts** the existing on-disk data (zvols/LVs + DRBD metadata) in place — no data is copied or resynced. This runbook is the required procedure; the converter is only one step of it. - -> **Scope.** This adopts a LINSTOR deployment whose satellites already run on the nodes blockstor will manage, keeping the same storage pools, DRBD minors, node-ids and (with the port capture below) TCP ports. It does not move data between nodes and does not change replica placement. - -## What the converter does and does not migrate - -Migrated: nodes, storage pools, resource groups, resource definitions (+ volume definitions), resources (replicas), and SUCCESSFUL snapshots. Controller-wide `DrbdOptions/*` become a `ControllerConfig`. DRBD minors, per-replica node-ids, the shared-secret and (when captured) TCP ports are preserved verbatim. An RD is stamped `Initialized=true` whenever its data still exists somewhere, and every replica gets `skipInitialSync=false`, so any replica added **after** migration performs a real sync instead of falsely coming up UpToDate. The latch is conditional, and it errs towards staying on: it is dropped only where no data-bearing replica is being adopted. That covers two shapes — every replica already DELETE-flagged, so nothing is migrating at all, and every replica diskless, so replicas are migrating but a client or quorum witness carries no copy to latch on. A definition whose replicas were held back because their nodes did not migrate keeps the latch, because that data is still on those nodes; so does one whose replicas were skipped for pool divergence. Unlatching in either of those would let the controller seed an empty first sync over data that exists. The report on stderr names every definition in these states, and names each reason that applies rather than only the first, so do not clear a latch by hand on the strength of a definition arriving without replicas. Snapshots are stamped `blockstor.io/adopted` so the controller records them as complete without re-taking them. - -**NOT migrated — read the report on stderr for the exact list per cluster:** - -- **LUKS passphrases.** LUKS-encrypted volumes convert with their layer stack intact, but the per-volume passphrase (encrypted with the LINSTOR master key) is not decrypted. You must provision `spec.encryption` for those RDs by hand before the encrypted volumes can be opened. Each affected RD is reported. -- **Backup-shipping remotes**, unknown/runtime flag bits, and DELETE'd / FAILED_DEPLOYMENT rows are skipped and reported. None affect live data availability. -- **DRBD TCP ports** are absent from most LINSTOR 1.33 dumps — see "Capture live DRBD ports" below. - -## Pre-flight (do all of these before touching anything) - -1. **Cluster is healthy.** `linstor r l --faulty` is empty; every resource is `UpToDate` on its diskful replicas; no node is `OFFLINE`. Do not migrate a degraded cluster — an Inconsistent/SyncTarget replica adopted as-is stays that way. - -2. **Take the database dump** (LINSTOR's k8s-backend CRDs): - - ```bash - mkdir dump && cd dump - kubectl get crds | grep -o ".*.internal.linstor.linbit.com" \ - | xargs -I{} sh -c 'kubectl get {} -ojson > {}.json' - cd .. - ``` - -3. **Capture the live DRBD ports.** LINSTOR 1.33 does not persist the port in its database, so the dump alone cannot preserve it. If blockstor adopts with a different port, `drbdadm adjust` moves the connection endpoint under live I/O (a reconnect blip, and on a marginal quorum a transient quorum loss). Capture the running ports from every satellite and union them: - - ```bash - for pod in $(kubectl -n cozy-linstor get pods -l app.kubernetes.io/component=linstor-satellite \ - -o jsonpath='{.items[*].metadata.name}'); do - kubectl -n cozy-linstor exec "$pod" -- sh -c ' - for f in /var/lib/linstor.d/*.res; do - rd=$(basename "$f" .res) - port=$(grep -oE "address[^;]*:[0-9]+" "$f" | grep -oE "[0-9]+$" | head -1) - [ -n "$port" ] && echo "$rd $port" - done' - done | sort -u > drbd-ports.txt - ``` - - Every replica of an RD shares one port, so duplicates across nodes are expected; `sort -u` collapses them. If an RD appears with two different ports, stop and investigate before proceeding. - -4. **Back up the ZFS/LVM pools' snapshots** (optional but recommended): a `zfs snapshot -r` of each pool gives an instant rollback point that is independent of both control planes. - -5. **Verify the zvol names blockstor will adopt match what is on disk.** Adoption is name-based: blockstor addresses each volume as `/_` (e.g. `data/pvc-abc…_00000`), and `CreateVolume` is idempotent only when that dataset already exists. A byte mismatch (ZFS names are case-sensitive) makes blockstor create a fresh EMPTY zvol next to the real one. Multi-replica DRBD would self-heal by SyncTarget, but **single-replica `["STORAGE"]` volumes have no resync fallback** — a mismatch there silently presents an empty disk. Cross-check before cutover, read-only, on each satellite: - - ```bash - # what is on disk: - zfs list -H -o name -t volume | sort > /tmp/ondisk-zvols.txt - # what blockstor will look for (from the converted manifests): - # /_ - # compare the two sets; every single-replica STORAGE-only volume MUST - # already exist on disk under the blockstor name. - ``` - - If any single-replica volume's computed name is absent from `ondisk-zvols.txt`, STOP — do not cut over until the naming is reconciled (`zfs rename`, or fix the converter). - - **Adopted snapshots are addressed the same way** — `/_00000@` — so run the same cross-check for them, otherwise a name miss only surfaces when someone tries to restore, long after cutover: - - ```bash - zfs list -H -o name -t snapshot | sort > /tmp/ondisk-snapshots.txt - # every migrated Snapshot's /_00000@ must appear here - ``` - -6. **Rehearse on staging first.** Do a full dry-run adoption of a COPY of the pools (or a representative subset) on a non-production cluster: apply the manifests, confirm every StoragePool registers a provider (no `provider requires … in props` errors in the satellite log), every Resource reaches `UpToDate` with `out-of-sync=0` (adopted, not resynced), and no new empty zvols appear. Only after a clean staging rehearsal proceed to production. - -## Convert - -```bash -linstor-migrate -in ./dump -drbd-ports ./drbd-ports.txt -out ./blockstor-manifests.yaml -``` - -Read the report on stderr end to end. Every `warning:` line is a thing the converter refused to guess — decide per line whether it matters for your data before continuing. In particular resolve every `LUKS passphrase NOT migrated` and every `no DRBD port` line (the latter should be empty if the port capture was complete). - -## Cutover - -The manifests do not migrate a cluster by themselves — the control plane must be swapped. Perform this in a maintenance window. - -1. **Freeze provisioning.** Scale the CSI provisioner/attacher to zero (or cordon new PVC creation) so no new volumes are created against the old controller mid-cutover. - -2. **Stop the LINSTOR controller.** Scale `deploy/linstor-controller` to 0. The satellites and the running DRBD devices keep serving I/O — only the control plane goes away. Do NOT stop the satellites or unload DRBD. - -3. **Deploy blockstor** (apiserver + controller + satellites) pointed at the same nodes and pools. The satellites must land on the same nodes with the same pool backing devices. Do not let blockstor create-md or mkfs — the adoption guards (existing DRBD metadata is not re-created, a populated filesystem is not re-formatted) protect the data, but verify the satellite image is the intended version first. - -4. **Apply the manifests in order.** `WriteManifests` already emits them in dependency order, so a single apply works: - - ```bash - kubectl apply -f ./blockstor-manifests.yaml - ``` - - If you prefer staged application, the order is: `ControllerConfig` → `Node` → `StoragePool` → `ResourceGroup` → `ResourceDefinition` → `Resource` → `Snapshot`. - -5. **Wait for adoption to converge.** Every `Resource` should report its DRBD device `UpToDate` (adopted, not resynced — watch that `out-of-sync` stays 0, i.e. no unexpected full sync started). `linstor r l --faulty` against the blockstor apiserver is empty. Storage pools report their real free capacity. - -6. **Repoint and unfreeze CSI.** Point linstor-csi at the blockstor apiserver, then scale the provisioner/attacher back up. Provision one test PVC and confirm it binds and that an existing PVC still mounts read-write. - -## Verify (before declaring done) - -- Object counts match the source: `linstor n l`, `sp l`, `rg l`, `rd l`, `r l` against blockstor equal the pre-cutover counts (the converter's report prints them). -- Spot-check ≥5 RDs: size, per-volume `/dev/drbd`, per-replica node-id and port match the pre-migration values. A changed minor or node-id is a stop-the-line defect. -- Every diskful replica is `UpToDate`; no replica is unexpectedly `SyncTarget`. -- Adopted snapshots list as present without any suspend-io having fired on the parent RDs (check the satellites did not log `suspend-io`). -- For LUKS RDs: their volumes stay closed until you provision `spec.encryption`; confirm consumers of those volumes are quiesced until then. - -## Rollback - -Until CSI is repointed and confirmed, rollback is: scale blockstor to 0, scale `deploy/linstor-controller` back to 1, repoint CSI at LINSTOR. The on-disk data and running DRBD devices were never touched, so this is a control-plane-only revert. If a pool-level `zfs snapshot` was taken in pre-flight, it is the last-resort data rollback point. - -## Known limitations (accept or mitigate explicitly) - -| Item | Impact | Mitigation | -| --- | --- | --- | -| LUKS passphrases not migrated | Encrypted volumes cannot be opened until `spec.encryption` is provisioned | Provision by hand; keep those consumers quiesced during migration | -| DRBD port not in dump (LINSTOR 1.33) | Adoption reconnects the mesh on a new port if not captured | Capture with the pre-flight command and pass `-drbd-ports` | -| Unknown flag bits dropped | Cosmetic runtime bookkeeping only | Reported; verified not to carry availability semantics | -| Placement/data unchanged | The converter never moves data | Rebalance with blockstor after migration if desired | -| `Initialized` is conditional | Left unlatched only where no data-bearing replica is adopted — every replica DELETE-flagged, or every replica diskless; a definition held back by an unmigrated node or by pool divergence keeps the latch and arrives short of replicas | Read the report, which names every reason that applies. Do not clear a latch by hand because a definition has no replicas — that is what re-enables the auto-primary election this latch exists to suppress | -| Single-replica zvol name must byte-match on disk | A mismatch presents an empty disk (no DRBD resync fallback) | Pre-flight step 5 cross-check + staging rehearsal (step 6) | -| Snapshots cover volume 0 only | blockstor addresses a snapshot as `/_00000@`; a snapshot that captured a multi-volume resource adopts with only its first volume restorable | Reported per snapshot by the converter; re-take those snapshots after migration if the other volumes matter | -| Undecoded flag bits are reported, not interpreted | A `non-zero vlm_flags` / `node_flags` / `unhandled flags bits` line means the source cluster had a marker this tool refuses to guess at (e.g. a size-semantics or eviction bit) | Look the object up in the source cluster before cutover (`linstor v l` / `n l`) and confirm it should be adopted as-is; the volume's `SizeKib` is carried verbatim either way | -| Node type mapping is empirically calibrated | Only `node_type=2` (SATELLITE) was confirmed against a live cluster; a CONTROLLER/AUXILIARY ordering error would adopt a controller node as AUXILIARY instead of skipping it | Controller nodes carry no pools or replicas, so impact is bounded — but check the report for unexpected node kinds | diff --git a/site/content/en/docs/migration.md b/site/content/en/docs/migration.md new file mode 100644 index 00000000..1d74a08c --- /dev/null +++ b/site/content/en/docs/migration.md @@ -0,0 +1,103 @@ +--- +title: "Migrating from LINSTOR" +linkTitle: "Migrating from LINSTOR" +weight: 70 +description: "Take over an existing LINSTOR cluster in place, without copying data." +--- + +Blockstor can take over a LINSTOR cluster that runs on Kubernetes. The `linstor-migrate` tool converts LINSTOR's database into blockstor resources, and blockstor then adopts the volumes and DRBD metadata that are already on disk. No data is copied or resynced, and replica placement does not change. + +Do this in a maintenance window, after a full rehearsal on a staging cluster. + +## What moves and what does not + +Converted: + +- nodes, storage pools, resource groups, resource definitions with their volumes, replicas, and successful snapshots; +- controller-wide DRBD options; +- DRBD minor numbers, node IDs and the shared secret, unchanged; +- DRBD ports, if you capture them from the running nodes (step 3 below). + +Not converted, and listed in the tool's report: + +- **LUKS passphrases.** Encrypted volumes convert with their layers intact, but blockstor cannot open them until you set up the passphrase by hand. +- Backup shipping remotes, and replicas or snapshots that LINSTOR marked as deleted or failed. +- Snapshots of definitions with more than one volume are adopted with their first volume only. + +## Before you start + +1. **Check that the cluster is healthy.** `linstor resource list --faulty` is empty, every replica with data is `UpToDate`, and no node is offline. A degraded replica is adopted as it is and stays degraded. + +2. **Dump the LINSTOR database.** LINSTOR on Kubernetes keeps it in custom resources: + + ```sh + mkdir dump && cd dump + kubectl get crds | grep -o ".*.internal.linstor.linbit.com" \ + | xargs -I{} sh -c 'kubectl get {} -ojson > {}.json' + cd .. + ``` + +3. **Capture the DRBD ports.** LINSTOR 1.33 does not store them in its database. Without them, blockstor would assign new ports, and every replica would reconnect under load. Collect them from the satellites: + + ```sh + for pod in $(kubectl -n cozy-linstor get pods -l app.kubernetes.io/component=linstor-satellite \ + -o jsonpath='{.items[*].metadata.name}'); do + kubectl -n cozy-linstor exec "$pod" -- sh -c ' + for f in /var/lib/linstor.d/*.res; do + rd=$(basename "$f" .res) + port=$(grep -oE "address[^;]*:[0-9]+" "$f" | grep -oE "[0-9]+$" | head -1) + [ -n "$port" ] && echo "$rd $port" + done' + done | sort -u > drbd-ports.txt + ``` + + Change the namespace and label to match your installation. Every replica of a volume shares one port. If one volume shows two different ports, stop and find out why. + +4. **Take pool snapshots.** A recursive `zfs snapshot` of every pool gives you a rollback point that does not depend on either control plane. + +5. **Check the volume names on disk.** Blockstor finds existing volumes by name: `/_`, for example `data/pvc-1234_00000`. If the name does not match exactly, blockstor creates a new empty volume next to the real one. Replicated volumes would heal from their peers, but a volume with a single replica and no DRBD would come up empty. On every node, compare the list of volumes with the names the converted manifests expect: + + ```sh + zfs list -H -o name -t volume | sort + zfs list -H -o name -t snapshot | sort + ``` + + Snapshots are found as `/_00000@`. Do not continue while any single-replica volume is missing under its expected name. + +6. **Rehearse on staging.** Run the whole procedure on a copy of the pools. Check that every pool registers, every replica reaches `UpToDate` without a resync, and no new empty volumes appear. + +## Convert + +Build the tool from the blockstor release you are installing and run it on the dump: + +```sh +go install github.com/cozystack/blockstor/cmd/linstor-migrate@v +linstor-migrate -in ./dump -drbd-ports ./drbd-ports.txt -out ./blockstor-manifests.yaml +``` + +Read the whole report it prints. Each `warning:` line is something the tool refused to guess. Resolve every `LUKS passphrase NOT migrated` and every `no DRBD port` line before you go on. + +## Cut over + +1. **Stop new volumes.** Scale the CSI provisioner and attacher to zero. +2. **Stop the LINSTOR controller.** Scale `deploy/linstor-controller` to zero. Leave the LINSTOR satellites and the DRBD devices running: they keep serving I/O. +3. **Install blockstor** as described in [Installation](../installation/), with satellites on the same nodes and pools. Check the image versions before you start them. +4. **Apply the converted manifests.** They are already in dependency order: + + ```sh + kubectl apply -f ./blockstor-manifests.yaml + ``` + +5. **Wait for adoption.** Every replica should report `UpToDate` without starting a resync, `blockstor resource list --faulty` should be empty, and every pool should report its real free space. +6. **Point CSI at blockstor**, as in [Provisioning volumes with CSI](../csi/), and scale the provisioner and attacher back up. Create a test PVC, and check that an existing PVC still mounts read-write. + +## Check before you call it done + +- The number of nodes, pools, groups, definitions and replicas matches the counts in the converter's report. +- For at least five volumes, the size, `/dev/drbd`, node IDs and port match the values from before the migration. A changed minor or node ID is a reason to stop and roll back. +- No replica is unexpectedly syncing. +- Encrypted volumes stay closed until you set up their passphrase. Keep their consumers stopped until then. + +## Roll back + +Until CSI has been pointed at blockstor and checked, rolling back means: scale blockstor to zero, scale `linstor-controller` back up, and point CSI at LINSTOR again. Blockstor did not touch the data on disk or the running DRBD devices, so this undoes only the control plane. The pool snapshots from step 4 are the last-resort rollback for the data itself. diff --git a/site/content/en/docs/overview.md b/site/content/en/docs/overview.md new file mode 100644 index 00000000..d8c9b43a --- /dev/null +++ b/site/content/en/docs/overview.md @@ -0,0 +1,57 @@ +--- +title: "Overview" +linkTitle: "Overview" +weight: 10 +description: "What blockstor is, what it does today, and when to use it." +--- + +Blockstor turns the local disks of your Kubernetes nodes into replicated block storage. Each volume is a logical volume (LVM) or a zvol (ZFS) on a node, and DRBD keeps copies of it on other nodes in sync, so a pod can lose its node and come back elsewhere with its data. + +Blockstor speaks the LINSTOR REST API. The clients built for LINSTOR, `linstor-csi`, `piraeus-operator`, `ha-controller` and `golinstor`, work against it unchanged. What is different is how the cluster keeps its state: everything lives in Kubernetes custom resources, and controllers converge the cluster toward it. There is no separate database to back up and no in-memory state to lose when a pod restarts. + +## When to use it + +Blockstor fits when you want replicated block volumes for Kubernetes on your own hardware, with LVM or ZFS underneath and DRBD for replication, and you want that storage to be managed the same way as the rest of the cluster: with custom resources, `kubectl` and GitOps. + +It is also a fit if you already run LINSTOR on Kubernetes. Blockstor can take over an existing LINSTOR cluster in place, without copying data. See [Migrating from LINSTOR](../migration/). + +## What works today + +- Replicated volumes over DRBD on LVM, LVM-thin, ZFS, ZFS-thin or file backends. +- Volumes without DRBD: plain local storage, single replica. +- LUKS encryption at rest. +- Automatic placement with constraints: zones, node properties, replicas on different nodes. +- Tiebreakers and quorum policies. +- Snapshots: create, restore as a new volume, and clone. +- Snapshot shipping between nodes of the same cluster, for clones and new replicas. +- Online volume resize. +- Creating storage pools from raw disks. +- The LINSTOR-compatible REST API over mutual TLS. + +## What is not implemented + +The API answers these with `501 Not Implemented`: + +- Snapshot shipping to another cluster (disaster recovery). +- Backups: create, restore, ship, abort, and the backup queue. +- Schedules (cron-driven backups). +- Remote backends such as S3 and LINSTOR remotes. +- Other storage providers: SPDK, NVMe-oF, OpenFlex, Exos. + +## On the roadmap + +- Bring-your-own-key encryption, with Secret references in the spec. +- Shared-LUN provisioning: thick LVM plus thin qcow2 on LVM. +- A VDUSE backend through `qemu-storage-daemon`, for shared-SAN Kubernetes. + +## Components + +Blockstor runs as three images, published to GHCR on every release: + +| Image | What it runs | +|---|---| +| `ghcr.io/cozystack/blockstor-controller` | The controllers that place replicas and allocate DRBD ports, minors and node IDs. | +| `ghcr.io/cozystack/blockstor-apiserver` | The LINSTOR-compatible REST API that CSI and other clients talk to. | +| `ghcr.io/cozystack/blockstor-satellite` | The per-node agent that creates volumes and runs DRBD. | + +[Architecture](../architecture/) shows how they fit together. diff --git a/site/content/en/docs/resources.md b/site/content/en/docs/resources.md new file mode 100644 index 00000000..60eb53ad --- /dev/null +++ b/site/content/en/docs/resources.md @@ -0,0 +1,201 @@ +--- +title: "Resources" +linkTitle: "Resources" +weight: 40 +description: "The blockstor custom resources, with examples." +--- + +Everything blockstor manages is a cluster-scoped custom resource in the `blockstor.cozystack.io/v1alpha1` API group. You can create and change them with `kubectl` or GitOps, with the [`blockstor` CLI](../cli/), or through the REST API. All three edit the same objects. + +| Kind | What it describes | Name | +|---|---|---| +| `Node` | A storage node and its replication address | the Kubernetes node name | +| `StoragePool` | A pool of space on one node | `.` | +| `PhysicalDevice` | A raw disk the satellite found on a node | set by the satellite | +| `ResourceGroup` | A placement and replication policy | any | +| `ResourceDefinition` | One volume set: sizes, layers, policy | any | +| `Resource` | One replica of a definition on one node | `.` | +| `Snapshot` | A snapshot of a definition | `.` | +| `ControllerConfig` | Cluster-wide settings | always `default` | + +The naming rules are enforced by the API server, so a `Resource` or `StoragePool` with a name that does not match its spec is refused. Each kind has printer columns, so `kubectl get resources` or `kubectl get storagepools` gives a useful table. + +Most objects have a `spec` you write and a `status` that blockstor writes. The examples below show only `spec`. + +## Node + +A node that runs the satellite. The address of the first network interface is the one DRBD uses for replication. + +```yaml +apiVersion: blockstor.cozystack.io/v1alpha1 +kind: Node +metadata: + name: worker-1 +spec: + type: SATELLITE + netInterfaces: + - name: default + address: 10.0.0.11 +``` + +## StoragePool + +Space for replicas on one node. The provider decides how volumes are allocated, and the properties name the backing volume group, thin pool, ZFS pool or directory. + +```yaml +apiVersion: blockstor.cozystack.io/v1alpha1 +kind: StoragePool +metadata: + name: data.worker-1 +spec: + nodeName: worker-1 + poolName: data + providerKind: LVM_THIN + props: + StorDriver/LvmVg: my_vg + StorDriver/ThinPool: my_thinpool +``` + +For a ZFS thin pool: + +```yaml +apiVersion: blockstor.cozystack.io/v1alpha1 +kind: StoragePool +metadata: + name: zfs-data.worker-1 +spec: + nodeName: worker-1 + poolName: zfs-data + providerKind: ZFS_THIN + props: + StorDriver/ZPoolThin: tank +``` + +The providers are `LVM`, `LVM_THIN`, `ZFS`, `ZFS_THIN`, `FILE`, `FILE_THIN` and `DISKLESS`. Pools with the same name on different nodes form one pool from the placement point of view: a policy that says "pool `data`" can place replicas on every node that has a `data` pool. + +## PhysicalDevice + +The satellites report the raw disks they find as `PhysicalDevice` objects, with their size, model and path in `status`. To build a pool from one, set `attachTo`: + +```yaml +apiVersion: blockstor.cozystack.io/v1alpha1 +kind: PhysicalDevice +metadata: + name: +spec: + attachTo: + storagePoolName: data + providerKind: ZFS_THIN + zPoolName: data +``` + +The CLI does the same thing with `blockstor physical-storage create-device-pool`. + +## ResourceGroup + +A policy that volumes inherit: how many replicas, in which pool, with which layers, and with which DRBD options. + +```yaml +apiVersion: blockstor.cozystack.io/v1alpha1 +kind: ResourceGroup +metadata: + name: replicated +spec: + description: Two data replicas plus a tiebreaker + selectFilter: + placeCount: 2 + storagePool: data + layerStack: ["DRBD", "STORAGE"] + replicasOnDifferent: ["Aux/topology.kubernetes.io/zone"] + drbdOptions: + resource: + quorum: majority + onNoQuorum: suspend-io + volumeGroups: + - volumeNumber: 0 +``` + +Placement constraints match node properties. Blockstor copies the Kubernetes node labels under `topology.kubernetes.io/` and `node-role.kubernetes.io/`, and `kubernetes.io/hostname`, onto each `Node` as `Aux/