diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index ff4f8eb99..bd28edd9c 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -11,3 +11,4 @@ - [ ] User-facing behavior and configuration docs are current - [ ] No credentials, private telemetry, host details, or enterprise-only source are included - [ ] API, migration, ingest, MCP/AG-UI, or release-contract changes are called out +- [ ] Breaking or batch-format changes carry `!` in the title, for the release's upgrade notes diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index f627e1f79..e689b2e06 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -60,6 +60,25 @@ jobs: exit 1 fi done <<< "${tags}" + # Before anything is published, so an unreadable batch format stops the + # release instead of leaving a format change unmarked. + - name: Write upgrade notes + env: + GH_TOKEN: ${{ github.token }} + run: | + set -euo pipefail + # Compare with the previous published release, not the previous tag: + # a tag whose release failed has no notes to carry its changes. + gh release list --repo "${GITHUB_REPOSITORY}" --exclude-drafts --limit 1000 \ + --json tagName --jq '.[].tagName' > "${RUNNER_TEMP}/published-tags" + bash scripts/release-notes.sh "${GITHUB_REF_NAME}" \ + < "${RUNNER_TEMP}/published-tags" > "${RUNNER_TEMP}/upgrade-notes.md" + cat "${RUNNER_TEMP}/upgrade-notes.md" + - uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: upgrade-notes + path: ${{ runner.temp }}/upgrade-notes.md + if-no-files-found: error - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 with: go-version-file: go.mod @@ -285,6 +304,10 @@ jobs: path: artifacts pattern: fanout_* merge-multiple: true + - uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2 + with: + name: upgrade-notes + path: notes - name: Checksum every archive run: | set -euo pipefail @@ -302,8 +325,11 @@ jobs: run: | set -euo pipefail # Maturity lives in the tag; keep this a full release so /releases/latest resolves. + # The upgrade notes from scripts/release-notes.sh are prepended to + # GitHub's generated changelog. gh release create "${GITHUB_REF_NAME}" \ --title "${GITHUB_REF_NAME}" \ + --notes "$(cat notes/upgrade-notes.md)" \ --generate-notes \ --verify-tag \ artifacts/fanout_*.tar.gz artifacts/SHA256SUMS diff --git a/docs/release.md b/docs/release.md index c3b3ba62c..44aaecc33 100644 --- a/docs/release.md +++ b/docs/release.md @@ -27,6 +27,38 @@ tags remain GHCR-only. Deployments should resolve public tags to immutable registry digests. Keep live manifests and secret profiles outside this public repository. +A tag without a suffix is a stable release; `-alpha`, `-beta` and `-rc` tags are +prereleases. Because `latest` follows both, it is not a stable channel. The +[upgrade policy](../site/src/content/docs/docs/guides/upgrade.mdx), published at +, sets the notice period for breaking +changes, the route and API deprecation rules, and the same-version restore +requirement. + +## Upgrade notes + +Every release's notes start with an **Upgrade notes** section that +`scripts/release-notes.sh` writes during the tag workflow's `verify` job, +before anything is published. It compares the tag with the previous +published, non-draft GitHub Release, which the workflow lists with +`gh release list`. A tag whose workflow failed has no release and is skipped, +so the next published release reports its changes. The section marks: + +- a batch format change, read from `batchMetadataVersion` in + `internal/telemetry/parquet.go`, as a **Format change** caution: the release + cannot read earlier telemetry; +- control migrations added under `internal/db/migrations`, which run forward + at startup, as a **Control schema migration**; +- otherwise, a **Storage** line stating that neither changed; +- merged changes whose titles carry the Conventional Commits `!` marker, under + **Breaking changes**. + +The script fails the release when it cannot read the format version, so a +format change is never published unmarked. `just release-test` covers it. The +section is prepended to GitHub's generated changelog; maintainers add manual +upgrade steps and deprecation notices below it. A format-changing or breaking +pull request should say so in its title with `!` and describe the operator +steps in its description, so they can be copied into the notes. + ## Creating a release Run `just release` from a clean, current `main`. The script: diff --git a/internal/telemetry/parquet.go b/internal/telemetry/parquet.go index 9bf660c8c..ca8b98ab7 100644 --- a/internal/telemetry/parquet.go +++ b/internal/telemetry/parquet.go @@ -26,8 +26,10 @@ import ( ) const ( - BatchSuffix = ".batch" - SchemaBatch = "_schema" + BatchSuffix + BatchSuffix = ".batch" + SchemaBatch = "_schema" + BatchSuffix + // scripts/release-notes.sh reads this declaration to mark format-changing + // releases; keep it a single integer literal. batchMetadataVersion = 5 parquetPageSize = 64 << 10 // A column whose distinct values outgrow this switches to plain pages for diff --git a/justfile b/justfile index e23720164..6e10573ef 100644 --- a/justfile +++ b/justfile @@ -123,14 +123,18 @@ docker TAG="local": # Tags are CalVer: vYYYY.M.N[-alpha|-beta|-rc]; N counts every release in the month. Pushing # the tag triggers release.yml, which builds the multi-architecture image, # mirrors it to Docker Hub, publishes native archives, and moves `latest`. +# scripts/release-notes.sh writes the release's upgrade notes, marking batch +# format changes, control migrations and `!` breaking changes. # Tag the next CalVer release and push it. release: ./scripts/release.sh -# Exercise reserved-tag retries without creating remote refs. +# Exercise reserved-tag retries without creating remote refs, and the upgrade +# notes that mark format-changing releases. release-test: bash ./scripts/release_test.sh + bash ./scripts/release-notes_test.sh # Caller tags must never replace the pinned native binding. duckdb-wrapper-test: diff --git a/scripts/release-notes.sh b/scripts/release-notes.sh new file mode 100755 index 000000000..0bade812c --- /dev/null +++ b/scripts/release-notes.sh @@ -0,0 +1,158 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Writes the "Upgrade notes" section that release.yml puts above GitHub's +# generated changelog. It compares the tag with the previous published release +# and marks, from the source rather than from memory: +# +# - a batch format change, which the new release cannot read across, so stored +# telemetry must be moved aside (internal/telemetry batchMetadataVersion); +# - new control database migrations, which run forward at startup, so rolling +# back means restoring a backup (internal/db/migrations); +# - breaking changes, from Conventional Commit subjects marked with `!`. +# +# Every release states its storage impact, so a missing marker is never +# mistaken for "no change". The script fails rather than guessing when the +# format version cannot be read. +# +# Usage: release-notes.sh < published-tags +# +# stdin lists the tags of published (non-draft) GitHub Releases, one per line. +# Git tags alone are not enough: a tag whose release workflow failed has no +# release, and comparing with it would hide a change it introduced. + +readonly RELEASE_TAG_PATTERN='^v([0-9]{4})\.([1-9]|1[0-2])\.(0|[1-9][0-9]*)(-(alpha|beta|rc))?$' +readonly FORMAT_SOURCE=internal/telemetry/parquet.go +readonly MIGRATIONS_DIR=internal/db/migrations +readonly UPGRADE_GUIDE=https://fanout.run/docs/guides/upgrade + +die() { + printf 'release-notes: %s\n' "$1" >&2 + exit 1 +} + +# Reads published release tags on stdin and prints the newest one older than $1. +previous_release_tag() { + local current=$1 + local tag + local best="" + local best_key=-1 + local key + local current_key + + [[ "$current" =~ $RELEASE_TAG_PATTERN ]] || die "release tag must match v{YYYY}.{M}.{N}[-alpha|-beta|-rc]: $current" + current_key=$(((10#${BASH_REMATCH[1]} * 100 + BASH_REMATCH[2]) * 1000000 + BASH_REMATCH[3])) + while IFS= read -r tag; do + [[ "$tag" =~ $RELEASE_TAG_PATTERN ]] || continue + key=$(((10#${BASH_REMATCH[1]} * 100 + BASH_REMATCH[2]) * 1000000 + BASH_REMATCH[3])) + if ((key < current_key && key > best_key)); then + best=$tag + best_key=$key + fi + done + printf '%s\n' "$best" +} + +# Prints the batch format version declared at a revision. +batch_format_at() { + local rev=$1 + local version + version=$(git show "${rev}:${FORMAT_SOURCE}" 2>/dev/null | + sed -n 's/^[[:space:]]*batchMetadataVersion[[:space:]]*=[[:space:]]*\([0-9][0-9]*\)[[:space:]]*$/\1/p') + [[ "$version" =~ ^[0-9]+$ ]] || die "cannot read batchMetadataVersion from ${FORMAT_SOURCE} at ${rev}" + printf '%s\n' "$version" +} + +# Prints the control migrations added between two revisions, one per line. +added_migrations() { + git diff --name-only --diff-filter=A "$1" "$2" -- "${MIGRATIONS_DIR}/*.sql" | + sed 's|.*/||' +} + +# Prints commit subjects marked as breaking (`type!:` or `type(scope)!:`). +breaking_subjects() { + git log --format=%s "$1..$2" | grep -E '^[a-z]+(\([^)]*\))?!:' || true +} + +# Renders the section from already-collected facts so it can be tested +# without a repository. +render_upgrade_notes() { + local previous=$1 + local old_format=$2 + local new_format=$3 + local migrations=$4 + local breaking=$5 + local count + + printf '## Upgrade notes\n\n' + if [ -z "$previous" ]; then + printf 'First release: there is no earlier release to compare storage formats with.\n' + return 0 + fi + + if [ "$old_format" != "$new_format" ]; then + printf '> [!CAUTION]\n' + printf '> **Format change: batch format %s to %s.** This release cannot read telemetry written by %s. ' "$old_format" "$new_format" "$previous" + printf 'Stored telemetry must be moved out of the data directory before starting it. ' + printf 'Follow [when a release changes the batch format](%s#when-a-release-changes-the-batch-format).\n\n' "$UPGRADE_GUIDE" + fi + + if [ -n "$migrations" ]; then + count=$(printf '%s\n' "$migrations" | wc -l | tr -d ' ') + printf '> [!IMPORTANT]\n' + if [ "$count" = 1 ]; then + printf '> **Control schema migration.** 1 new control database migration runs forward at startup. ' + else + printf '> **Control schema migration.** %s new control database migrations run forward at startup. ' "$count" + fi + printf 'Take a cold backup first: rolling back means restoring that backup with %s. ' "$previous" + printf 'See [upgrade an instance](%s#upgrade-an-instance).\n>\n' "$UPGRADE_GUIDE" + # shellcheck disable=SC2016 # Literal Markdown backticks. + printf '%s\n' "$migrations" | sed 's/^/> - `/; s/$/`/' + printf '\n' + fi + + if [ "$old_format" = "$new_format" ] && [ -z "$migrations" ]; then + printf '**Storage:** batch format %s is unchanged and no control database migrations were added since %s.\n\n' "$new_format" "$previous" + fi + + if [ -n "$breaking" ]; then + printf '### Breaking changes\n\n' + printf '%s\n' "$breaking" | sed 's/^/- /' + printf '\n' + fi +} + +main() { + local tag=${1:-} + local previous + local old_format="" + local new_format + local migrations="" + local breaking="" + + [ -n "$tag" ] || die "usage: release-notes.sh < published-tags" + git rev-parse --verify --quiet "refs/tags/${tag}" >/dev/null || die "tag not found: $tag" + previous=$(previous_release_tag "$tag") || exit 1 + if [ -n "$previous" ]; then + git rev-parse --verify --quiet "refs/tags/${previous}" >/dev/null || + die "previous release tag is not fetched: $previous" + else + # "First release" is only true when no older release tag exists at all. An + # empty or truncated release list must not hide every earlier change. + local older + older=$(git tag --list 'v*.*.*' | previous_release_tag "$tag") || exit 1 + [ -z "$older" ] || die "no published release precedes $tag, but tag $older does; check the release list" + fi + new_format=$(batch_format_at "$tag") || exit 1 + if [ -n "$previous" ]; then + old_format=$(batch_format_at "$previous") || exit 1 + migrations=$(added_migrations "$previous" "$tag") || exit 1 + breaking=$(breaking_subjects "$previous" "$tag") || exit 1 + fi + render_upgrade_notes "$previous" "$old_format" "$new_format" "$migrations" "$breaking" +} + +if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then + main "$@" +fi diff --git a/scripts/release-notes_test.sh b/scripts/release-notes_test.sh new file mode 100644 index 000000000..27a4fe505 --- /dev/null +++ b/scripts/release-notes_test.sh @@ -0,0 +1,138 @@ +#!/usr/bin/env bash +set -euo pipefail + +# shellcheck source=scripts/release-notes.sh +source "$(dirname "$0")/release-notes.sh" + +fail() { + printf 'release-notes_test: %s\n' "$1" >&2 + exit 1 +} + +contains() { + [[ "$1" == *"$2"* ]] || fail "$3: output does not contain '$2':"$'\n'"$1" +} + +lacks() { + [[ "$1" != *"$2"* ]] || fail "$3: output unexpectedly contains '$2':"$'\n'"$1" +} + +test_previous_release_tag() { + local got + got=$(previous_release_tag v2026.11.0-alpha <<< $'v2026.10.9-rc\nv2026.10.24-alpha\nv2026.10.100-preview\nv2026.11.0-alpha\nv2026.11.1-alpha\nv2025.12.99') + [ "$got" = v2026.10.24-alpha ] || fail "previous of v2026.11.0-alpha = $got" + got=$(previous_release_tag v2027.1.0 <<< $'v2026.12.3-rc\nv2026.9.40') + [ "$got" = v2026.12.3-rc ] || fail "previous across a year = $got" + got=$(previous_release_tag v2026.11.0-alpha <<< $'v2026.11.0-alpha\nnot-a-tag') + [ -z "$got" ] || fail "first release has a previous tag: $got" + if (previous_release_tag v2026.11.0-preview <<< '' 2>/dev/null); then + fail "an invalid tag was accepted" + fi +} + +test_render() { + local out + out=$(render_upgrade_notes v2026.10.24-alpha 4 5 $'a.sql\nb.sql' 'feat!: rename a route (#1)') + contains "$out" "**Format change: batch format 4 to 5.**" format-change + contains "$out" "cannot read telemetry written by v2026.10.24-alpha" format-change + contains "$out" "#when-a-release-changes-the-batch-format" format-change + contains "$out" "2 new control database migrations run forward" migrations + contains "$out" "> - \`b.sql\`" migrations + contains "$out" "- feat!: rename a route (#1)" breaking + lacks "$out" "**Storage:**" format-change + + out=$(render_upgrade_notes v2026.11.1-alpha 5 5 'a.sql' '') + contains "$out" "1 new control database migration runs forward" one-migration + lacks "$out" "Format change" one-migration + lacks "$out" "Breaking changes" one-migration + + out=$(render_upgrade_notes v2026.11.1-alpha 5 5 '' '') + contains "$out" "**Storage:** batch format 5 is unchanged and no control database migrations were added since v2026.11.1-alpha." unchanged + + out=$(render_upgrade_notes '' '' 5 '' '') + contains "$out" "First release" first +} + +# Builds a throwaway repository with two releases and runs the script end to end. +test_main() ( + local repo + local out + # Git hooks export GIT_DIR and friends; without clearing them, `git init` + # below would reinitialize the repository running the hook. + # shellcheck disable=SC2046 # One variable name per word. + unset $(git rev-parse --local-env-vars) + # Nor may a contributor's global or system configuration, hooks or templates + # take part. + export GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_NOSYSTEM=1 + repo=$(mktemp -d) + trap 'rm -rf "$repo"' EXIT + cd "$repo" + if git rev-parse --git-dir >/dev/null 2>&1; then + fail "the temporary directory is already inside a repository" + fi + g() { git -c user.name=test -c user.email=test@example.com -c commit.gpgsign=false -c tag.gpgsign=false "$@"; } + g init -q + mkdir -p internal/telemetry internal/db/migrations + printf 'const (\n\tbatchMetadataVersion = 4\n)\n' > internal/telemetry/parquet.go + printf -- '-- +goose Up\n' > internal/db/migrations/20260101000000_init.sql + g add . + g commit -qm 'feat: initial' + g tag v2026.1.0-alpha + printf 'const (\n\tbatchMetadataVersion = 5\n)\n' > internal/telemetry/parquet.go + g commit -qam 'feat: batch format 5 (#2)' + printf -- '-- +goose Up\n' > internal/db/migrations/20260201000000_next.sql + g add . + g commit -qm 'refactor(api)!: rename routes (#3)' + g tag v2026.2.0-alpha + g commit -q --allow-empty -m 'fix: unrelated (#4)' + g tag v2026.2.1-alpha + + local published=$'v2026.1.0-alpha\nv2026.2.0-alpha\nv2026.2.1-alpha' + out=$(main v2026.2.0-alpha <<< "$published") + contains "$out" "**Format change: batch format 4 to 5.**" main-format + contains "$out" "\`20260201000000_next.sql\`" main-migration + lacks "$out" "20260101000000_init.sql" main-migration + contains "$out" "- refactor(api)!: rename routes (#3)" main-breaking + lacks "$out" "fix: unrelated" main-breaking + + out=$(main v2026.2.1-alpha <<< "$published") + contains "$out" "**Storage:** batch format 5 is unchanged" main-unchanged + + out=$(main v2026.1.0-alpha <<< "$published") + contains "$out" "First release" main-first + out=$(main v2026.1.0-alpha < /dev/null) + contains "$out" "First release" main-first-empty-list + + # An empty release list with older tags in git is an API failure, not a + # first release. + if (main v2026.2.1-alpha < /dev/null >/dev/null 2>&1); then + fail "an empty release list was reported as a first release" + fi + + # v2026.2.0-alpha was tagged but its release failed, so it was never + # published: v2026.2.1-alpha must carry its format change and migration. + out=$(main v2026.2.1-alpha <<< $'v2026.1.0-alpha\nv2026.2.1-alpha') + contains "$out" "**Format change: batch format 4 to 5.**" main-unpublished + contains "$out" "cannot read telemetry written by v2026.1.0-alpha" main-unpublished + contains "$out" "\`20260201000000_next.sql\`" main-unpublished + contains "$out" "- refactor(api)!: rename routes (#3)" main-unpublished + lacks "$out" "**Storage:**" main-unpublished + + # A published tag that was not fetched must stop the release. + if (main v2026.2.1-alpha <<< $'v2026.1.5-alpha\nv2026.2.1-alpha' >/dev/null 2>&1); then + fail "a missing previous tag was accepted" + fi + + # A renamed or missing constant must stop the release, not hide a format change. + printf 'const (\n\tbatchFormat = 6\n)\n' > internal/telemetry/parquet.go + g commit -qam 'refactor: rename the constant (#5)' + g tag v2026.3.0-alpha + if (main v2026.3.0-alpha <<< "$published" >/dev/null 2>&1); then + fail "an unreadable format version was accepted" + fi +) + +test_previous_release_tag +test_render +test_main +printf 'release-notes_test: all tests passed\n' diff --git a/site/src/content/docs/docs/guides/upgrade.mdx b/site/src/content/docs/docs/guides/upgrade.mdx index 20b4cc674..8a18b34a2 100644 --- a/site/src/content/docs/docs/guides/upgrade.mdx +++ b/site/src/content/docs/docs/guides/upgrade.mdx @@ -1,31 +1,144 @@ --- title: Upgrade -description: Replace a Fanout release and handle storage format changes. -summary: Back up first, retain control state, and follow the target release notes. +description: Which Fanout releases are stable, how breaking and format-changing releases are announced, and how to replace a release. +summary: Pin a version, read every release's upgrade notes, back up first, and restore only with the version that made the backup. read_when: - You are upgrading an existing instance. + - You are choosing which releases to run in production. + - You maintain scripts or clients against Fanout's HTTP routes, MCP tools or settings. status: preview --- +## Release channels + +Fanout versions are CalVer tags, `vYYYY.M.N` with an optional maturity suffix. +`N` counts every release in the month, whatever its suffix. Container image +tags omit the leading `v`: release `v2026.9.5` is image tag `2026.9.5`. + +| Tag | Channel | Example | +|---|---|---| +| No suffix | Stable | `v2026.9.5` | +| `-rc` | Prerelease: release candidate | `v2026.9.4-rc` | +| `-beta` | Prerelease | `v2026.9.2-beta` | +| `-alpha` | Prerelease | `v2026.9.0-alpha` | + +The tag of each +[GitHub Release](https://github.com/labstack/fanout/releases) shows its channel. + +Every release, stable or prerelease, is a full GitHub Release with the same +contents: Linux and macOS archives for amd64 and arm64, a `SHA256SUMS` file, +GitHub build-provenance attestations for each archive and for `SHA256SUMS`, and +a multi-architecture image on GHCR with a Docker Hub mirror of the same digest. +[Releases](https://github.com/labstack/fanout/blob/main/docs/release.md#consumer-verification) +shows how to verify them. Fix a problem in a published version with a new +release number, never by replacing that version. A version's image tag can +still be pushed again, so record the digest you deploy and compare it with the +one published for the release. + +The `latest` image tag and the installer's default follow the newest release +of either channel. They are not a stable channel. To run stable releases only, +pin a release tag without a suffix (image tag `2026.9.5` for `v2026.9.5`), or +the image digest it resolves to, and upgrade deliberately. + +## Read the upgrade notes + +Each GitHub Release starts with an **Upgrade notes** section. The release +workflow writes it by comparing the tag with the previous published release. +A tag whose release workflow failed has no release and is skipped, so its +changes are reported by the next published release: + +- **Format change: batch format X to Y** (a caution callout) marks a release + that cannot read telemetry written by earlier releases. Follow + [when a release changes the batch format](#when-a-release-changes-the-batch-format). +- **Control schema migration** lists the control database migrations the + release adds. They run forward at startup, so rolling back means restoring a + backup. +- **Storage** states that the batch format is unchanged and no migrations were + added, so a missing marker is never ambiguous. +- **Breaking changes** lists the merged changes whose titles carry the + Conventional Commits `!` marker, for example `feat(auth)!:`. + +Maintainers add any manual steps below these markers. The markers compare each +release only with the one before it. When you skip releases, read the notes of +every release between your version and the target. + +A release without a generated section predates it. Its notes are +handwritten; read them for the same changes. For example, +[v2026.11.0-alpha](https://github.com/labstack/fanout/releases/tag/v2026.11.0-alpha) +changes the batch format from 4 to 5. + +## Breaking changes and notice + +A breaking change is one that needs action from an operator or client after +upgrading. That includes removing or renaming an HTTP route, a JSON field or +query parameter, an MCP tool or its arguments, or a configuration variable; +changing a credential format; and changing the batch format. + +- **Prereleases** may ship a breaking change in any release, without notice. + The release's upgrade notes mark it. +- **Stable releases** announce a breaking change in the notes of an earlier + stable release at least 30 days before a stable release ships it. The + announcement names what changes, its replacement, and the earliest release + that may change it. + +Additions are not breaking: new routes, tools, optional fields and settings +can appear in any release. + +## Route and API deprecation + +Fanout does not serve an old and a new name side by side. A rename or removal +changes the route registration, its clients, its authorization and the +generated [HTTP routes](/docs/reference/api) and [MCP tools](/docs/reference/mcp) +reference in one release, with no alias or fallback. + +Deprecation is therefore a notice, not a transition period: + +1. A stable release's upgrade notes announce the deprecated route, field, tool + or setting, with its replacement and the earliest release that removes it. +2. Until that release, the deprecated contract keeps working unchanged. +3. The removing release lists the change under **Breaking changes**. + +Protocol-defined contracts follow their own specifications rather than this +schedule: OTLP ingest, the MCP protocol, OAuth and the `/.well-known` metadata +paths. Product CalVer never appears in API paths or MCP tool names. + ## Upgrade an instance 1. Read the target [GitHub Release](https://github.com/labstack/fanout/releases) - and pin its binary or image tag. Check for batch-format and API changes. -2. Take and verify a [cold backup](/docs/guides/backup-and-restore). + and every release you are skipping, and pin the target's binary or image + tag. +2. Take and verify a [cold backup](/docs/guides/backup-and-restore). Record the + running version with it. 3. Stop Fanout and wait for it to exit. 4. Replace the binary or image. Keep the data directory unless the release requires the format-change procedure below. 5. Start Fanout. Check `/readyz`, sign-in, configured OTLP transports and representative queries. -Pending control SQLite migrations run forward at startup. A rollback after -migration is not assumed safe. Restore the complete pre-upgrade backup with -the previous binary when rollback is required. +Pending control SQLite migrations run forward, in order, at startup, so one +upgrade can cross several releases. Migrations have no down path. A rollback +after migration is not assumed safe. + +## Roll back or restore + +Restore a backup only with **the version that created it**. Fanout does not +record which release wrote a data directory, so it cannot warn about a +mismatch: + +- A release reads only its own batch format and refuses to start on any other. +- An earlier release is not designed to run against a control database that a + later release migrated. + +To roll back, stop Fanout, restore the complete pre-upgrade backup, and start +the previous binary or image. To move to a newer version after a restore, +verify the restored instance first, then upgrade with the steps above. See +[back up and restore](/docs/guides/backup-and-restore). ## When a release changes the batch format [v2026.11.0-alpha](https://github.com/labstack/fanout/releases/tag/v2026.11.0-alpha) -requires batch format 5 and cannot read format 4. For that upgrade: +requires batch format 5 and cannot read format 4. For that upgrade, and any +release whose notes carry a **Format change** marker: 1. Stop Fanout and take a complete cold backup. 2. Move `telemetry/` and `query/` out of the data directory. Keep them together @@ -40,8 +153,8 @@ Old telemetry is unavailable in the upgraded instance. Query state rebuilds from the new telemetry. Do not copy unsupported batches back into it. See [storage](/docs/concepts/storage) for the current format contract. -This release also renames API and MCP contracts without aliases. Update scripts -and clients using the release notes and [HTTP routes](/docs/reference/api). +v2026.11.0-alpha also renames API and MCP contracts without aliases. Update +scripts and clients using the release notes and [HTTP routes](/docs/reference/api). ## When a release changes credential formats