Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,4 @@
- [ ] User-facing behavior and configuration docs are current
- [ ] No credentials, private telemetry, host details, or enterprise-only source are included
- [ ] API, migration, ingest, MCP/AG-UI, or release-contract changes are called out
- [ ] Breaking or batch-format changes carry `!` in the title, for the release's upgrade notes
26 changes: 26 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,25 @@ jobs:
exit 1
fi
done <<< "${tags}"
# Before anything is published, so an unreadable batch format stops the
# release instead of leaving a format change unmarked.
- name: Write upgrade notes
env:
GH_TOKEN: ${{ github.token }}
run: |
set -euo pipefail
# Compare with the previous published release, not the previous tag:
# a tag whose release failed has no notes to carry its changes.
gh release list --repo "${GITHUB_REPOSITORY}" --exclude-drafts --limit 1000 \
--json tagName --jq '.[].tagName' > "${RUNNER_TEMP}/published-tags"
bash scripts/release-notes.sh "${GITHUB_REF_NAME}" \
< "${RUNNER_TEMP}/published-tags" > "${RUNNER_TEMP}/upgrade-notes.md"
cat "${RUNNER_TEMP}/upgrade-notes.md"
- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: upgrade-notes
path: ${{ runner.temp }}/upgrade-notes.md
if-no-files-found: error
- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
with:
go-version-file: go.mod
Expand Down Expand Up @@ -285,6 +304,10 @@ jobs:
path: artifacts
pattern: fanout_*
merge-multiple: true
- uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
name: upgrade-notes
path: notes
- name: Checksum every archive
run: |
set -euo pipefail
Expand All @@ -302,8 +325,11 @@ jobs:
run: |
set -euo pipefail
# Maturity lives in the tag; keep this a full release so /releases/latest resolves.
# The upgrade notes from scripts/release-notes.sh are prepended to
# GitHub's generated changelog.
gh release create "${GITHUB_REF_NAME}" \
--title "${GITHUB_REF_NAME}" \
--notes "$(cat notes/upgrade-notes.md)" \
--generate-notes \
--verify-tag \
artifacts/fanout_*.tar.gz artifacts/SHA256SUMS
Expand Down
32 changes: 32 additions & 0 deletions docs/release.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,38 @@ tags remain GHCR-only.
Deployments should resolve public tags to immutable registry digests. Keep live
manifests and secret profiles outside this public repository.

A tag without a suffix is a stable release; `-alpha`, `-beta` and `-rc` tags are
prereleases. Because `latest` follows both, it is not a stable channel. The
[upgrade policy](../site/src/content/docs/docs/guides/upgrade.mdx), published at
<https://fanout.run/docs/guides/upgrade>, sets the notice period for breaking
changes, the route and API deprecation rules, and the same-version restore
requirement.

## Upgrade notes

Every release's notes start with an **Upgrade notes** section that
`scripts/release-notes.sh` writes during the tag workflow's `verify` job,
before anything is published. It compares the tag with the previous
published, non-draft GitHub Release, which the workflow lists with
`gh release list`. A tag whose workflow failed has no release and is skipped,
so the next published release reports its changes. The section marks:

- a batch format change, read from `batchMetadataVersion` in
`internal/telemetry/parquet.go`, as a **Format change** caution: the release
cannot read earlier telemetry;
- control migrations added under `internal/db/migrations`, which run forward
at startup, as a **Control schema migration**;
- otherwise, a **Storage** line stating that neither changed;
- merged changes whose titles carry the Conventional Commits `!` marker, under
**Breaking changes**.

The script fails the release when it cannot read the format version, so a
format change is never published unmarked. `just release-test` covers it. The
section is prepended to GitHub's generated changelog; maintainers add manual
upgrade steps and deprecation notices below it. A format-changing or breaking
pull request should say so in its title with `!` and describe the operator
steps in its description, so they can be copied into the notes.

## Creating a release

Run `just release` from a clean, current `main`. The script:
Expand Down
6 changes: 4 additions & 2 deletions internal/telemetry/parquet.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,10 @@ import (
)

const (
BatchSuffix = ".batch"
SchemaBatch = "_schema" + BatchSuffix
BatchSuffix = ".batch"
SchemaBatch = "_schema" + BatchSuffix
// scripts/release-notes.sh reads this declaration to mark format-changing
// releases; keep it a single integer literal.
batchMetadataVersion = 5
parquetPageSize = 64 << 10
// A column whose distinct values outgrow this switches to plain pages for
Expand Down
6 changes: 5 additions & 1 deletion justfile
Original file line number Diff line number Diff line change
Expand Up @@ -123,14 +123,18 @@ docker TAG="local":
# Tags are CalVer: vYYYY.M.N[-alpha|-beta|-rc]; N counts every release in the month. Pushing
# the tag triggers release.yml, which builds the multi-architecture image,
# mirrors it to Docker Hub, publishes native archives, and moves `latest`.
# scripts/release-notes.sh writes the release's upgrade notes, marking batch
# format changes, control migrations and `!` breaking changes.

# Tag the next CalVer release and push it.
release:
./scripts/release.sh

# Exercise reserved-tag retries without creating remote refs.
# Exercise reserved-tag retries without creating remote refs, and the upgrade
# notes that mark format-changing releases.
release-test:
bash ./scripts/release_test.sh
bash ./scripts/release-notes_test.sh

# Caller tags must never replace the pinned native binding.
duckdb-wrapper-test:
Expand Down
158 changes: 158 additions & 0 deletions scripts/release-notes.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,158 @@
#!/usr/bin/env bash
set -euo pipefail

# Writes the "Upgrade notes" section that release.yml puts above GitHub's
# generated changelog. It compares the tag with the previous published release
# and marks, from the source rather than from memory:
#
# - a batch format change, which the new release cannot read across, so stored
# telemetry must be moved aside (internal/telemetry batchMetadataVersion);
# - new control database migrations, which run forward at startup, so rolling
# back means restoring a backup (internal/db/migrations);
# - breaking changes, from Conventional Commit subjects marked with `!`.
#
# Every release states its storage impact, so a missing marker is never
# mistaken for "no change". The script fails rather than guessing when the
# format version cannot be read.
#
# Usage: release-notes.sh <tag> < published-tags
#
# stdin lists the tags of published (non-draft) GitHub Releases, one per line.
# Git tags alone are not enough: a tag whose release workflow failed has no
# release, and comparing with it would hide a change it introduced.

readonly RELEASE_TAG_PATTERN='^v([0-9]{4})\.([1-9]|1[0-2])\.(0|[1-9][0-9]*)(-(alpha|beta|rc))?$'
readonly FORMAT_SOURCE=internal/telemetry/parquet.go
readonly MIGRATIONS_DIR=internal/db/migrations
readonly UPGRADE_GUIDE=https://fanout.run/docs/guides/upgrade

die() {
printf 'release-notes: %s\n' "$1" >&2
exit 1
}

# Reads published release tags on stdin and prints the newest one older than $1.
previous_release_tag() {
local current=$1
local tag
local best=""
local best_key=-1
local key
local current_key

[[ "$current" =~ $RELEASE_TAG_PATTERN ]] || die "release tag must match v{YYYY}.{M}.{N}[-alpha|-beta|-rc]: $current"
current_key=$(((10#${BASH_REMATCH[1]} * 100 + BASH_REMATCH[2]) * 1000000 + BASH_REMATCH[3]))
while IFS= read -r tag; do
[[ "$tag" =~ $RELEASE_TAG_PATTERN ]] || continue
key=$(((10#${BASH_REMATCH[1]} * 100 + BASH_REMATCH[2]) * 1000000 + BASH_REMATCH[3]))
if ((key < current_key && key > best_key)); then
best=$tag
best_key=$key
fi
done
printf '%s\n' "$best"
}

# Prints the batch format version declared at a revision.
batch_format_at() {
local rev=$1
local version
version=$(git show "${rev}:${FORMAT_SOURCE}" 2>/dev/null |
sed -n 's/^[[:space:]]*batchMetadataVersion[[:space:]]*=[[:space:]]*\([0-9][0-9]*\)[[:space:]]*$/\1/p')
[[ "$version" =~ ^[0-9]+$ ]] || die "cannot read batchMetadataVersion from ${FORMAT_SOURCE} at ${rev}"
printf '%s\n' "$version"
}

# Prints the control migrations added between two revisions, one per line.
added_migrations() {
git diff --name-only --diff-filter=A "$1" "$2" -- "${MIGRATIONS_DIR}/*.sql" |
sed 's|.*/||'
}

# Prints commit subjects marked as breaking (`type!:` or `type(scope)!:`).
breaking_subjects() {
git log --format=%s "$1..$2" | grep -E '^[a-z]+(\([^)]*\))?!:' || true
}

# Renders the section from already-collected facts so it can be tested
# without a repository.
render_upgrade_notes() {
local previous=$1
local old_format=$2
local new_format=$3
local migrations=$4
local breaking=$5
local count

printf '## Upgrade notes\n\n'
if [ -z "$previous" ]; then
printf 'First release: there is no earlier release to compare storage formats with.\n'
return 0
fi

if [ "$old_format" != "$new_format" ]; then
printf '> [!CAUTION]\n'
printf '> **Format change: batch format %s to %s.** This release cannot read telemetry written by %s. ' "$old_format" "$new_format" "$previous"
printf 'Stored telemetry must be moved out of the data directory before starting it. '
printf 'Follow [when a release changes the batch format](%s#when-a-release-changes-the-batch-format).\n\n' "$UPGRADE_GUIDE"
fi

if [ -n "$migrations" ]; then
count=$(printf '%s\n' "$migrations" | wc -l | tr -d ' ')
printf '> [!IMPORTANT]\n'
if [ "$count" = 1 ]; then
printf '> **Control schema migration.** 1 new control database migration runs forward at startup. '
else
printf '> **Control schema migration.** %s new control database migrations run forward at startup. ' "$count"
fi
printf 'Take a cold backup first: rolling back means restoring that backup with %s. ' "$previous"
printf 'See [upgrade an instance](%s#upgrade-an-instance).\n>\n' "$UPGRADE_GUIDE"
# shellcheck disable=SC2016 # Literal Markdown backticks.
printf '%s\n' "$migrations" | sed 's/^/> - `/; s/$/`/'
printf '\n'
fi

if [ "$old_format" = "$new_format" ] && [ -z "$migrations" ]; then
printf '**Storage:** batch format %s is unchanged and no control database migrations were added since %s.\n\n' "$new_format" "$previous"
fi

if [ -n "$breaking" ]; then
printf '### Breaking changes\n\n'
printf '%s\n' "$breaking" | sed 's/^/- /'
printf '\n'
fi
}

main() {
local tag=${1:-}
local previous
local old_format=""
local new_format
local migrations=""
local breaking=""

[ -n "$tag" ] || die "usage: release-notes.sh <tag> < published-tags"
git rev-parse --verify --quiet "refs/tags/${tag}" >/dev/null || die "tag not found: $tag"
previous=$(previous_release_tag "$tag") || exit 1
if [ -n "$previous" ]; then
git rev-parse --verify --quiet "refs/tags/${previous}" >/dev/null ||
die "previous release tag is not fetched: $previous"
else
# "First release" is only true when no older release tag exists at all. An
# empty or truncated release list must not hide every earlier change.
local older
older=$(git tag --list 'v*.*.*' | previous_release_tag "$tag") || exit 1
[ -z "$older" ] || die "no published release precedes $tag, but tag $older does; check the release list"
fi
new_format=$(batch_format_at "$tag") || exit 1
if [ -n "$previous" ]; then
old_format=$(batch_format_at "$previous") || exit 1
migrations=$(added_migrations "$previous" "$tag") || exit 1
breaking=$(breaking_subjects "$previous" "$tag") || exit 1
fi
render_upgrade_notes "$previous" "$old_format" "$new_format" "$migrations" "$breaking"
}

if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then
main "$@"
fi
Loading
Loading