Skip to content
CI

Pin the formatter, and stop format-gating generated migrations (#267) #519

Sign in to view logs

Pin the formatter, and stop format-gating generated migrations (#267)

Pin the formatter, and stop format-gating generated migrations (#267) #519

Workflow file for this run

.github/workflows/ci.yml at 1a7b60a
name: CI
on:
pull_request:
push:
branches: [main]
# Callable, so publishing a release runs these same checks against the release commit rather than
# trusting that they ran somewhere earlier.
workflow_call:
# Least privilege by default. Jobs that need more must declare it locally.
permissions:
contents: read
# The newest push on a branch wins; main runs are retained for badge and release history.
concurrency:
group: ci-${{ github.ref }}
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
jobs:
# Formatting, linting, typing and tests are independent checks, so each reports its own result.
static:
name: format, lint, types
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
# Checkout leaves a usable credential in the runner otherwise, which every later step and
# every action it calls can read. Nothing here pushes.
persist-credentials: false
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
with:
bun-version: 1.3.14
- run: bun install --frozen-lockfile
- run: bun run format:check
- run: bun run lint
- run: bun run typecheck
# The two packages that are deployables in their own right rather than root workspaces. Root
# `typecheck` is `bun run --filter '*' typecheck`, and `--filter '*'` enumerates `workspaces`, which
# is app, server and worker. So both of these ship a `typecheck` script that nothing has ever run:
# `agent-computer` holds the only `spawn` in the deployment and `supervisor` is the only thing
# holding a Docker socket, which is a poor pair to leave untyped. Both are clean today, so this
# changes nothing about main and only stops the next change being the first one a type checker sees.
#
# Not by adding them to `workspaces`: each is built from its own lockfile and the Dockerfile depends
# on that, so they are installed separately here for the same reason they are installed separately
# there. A matrix rather than two jobs, so each package reports its own result and a third one is a
# line in the list below and nothing in `verify`, which sees the whole matrix as one entry.
deployables:
name: types (${{ matrix.package }})
runs-on: ubuntu-latest
strategy:
# One red package must not hide whether the other is red too.
fail-fast: false
matrix:
package: [agent-computer, supervisor]
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
with:
bun-version: 1.3.14
# Two installs, and the root one is load-bearing. Both packages set `types: ["bun"]` in
# their tsconfig while `@types/bun` is a root devDependency, so tsc resolves it by walking up
# to the root node_modules. Without it the typecheck fails with TS2688 on a fresh checkout.
# This is the order the image uses for the same reason: Dockerfile installs at the root
# before installing agent-computer.
- run: bun install --frozen-lockfile
- run: bun install --frozen-lockfile
working-directory: ${{ matrix.package }}
- run: bun run typecheck
working-directory: ${{ matrix.package }}
chart:
name: chart (${{ matrix.target }})
runs-on: ubuntu-latest
strategy:
# One red target must not hide whether another is red too.
fail-fast: false
matrix:
target: [self-hosted, eks, eks-sandbox, gke, aks]
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: azure/setup-helm@9bc31f4ebc9c6b171d7bfbaa5d006ae7abdb4310 # v5.0.1
with:
version: v3.19.0
# For the coherence check below, which is a Bun script like everything else here.
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
with:
bun-version: 1.3.14
# Nothing rendered this chart until now, which is how four values files that produce a server
# unable to start were shipped and stayed shipped. Rendering is the cheap half; the refusals in
# validation.yaml are the half that catches a missing value before a cluster does.
- run: helm dependency build charts/openbot
# Structure only. `helm lint` reports a template `fail` as an INFO line and exits 0 even under
# `--strict`, which was driven and confirmed, so it cannot gate the refusals below. Rendering
# does: `helm template` exits non-zero on one.
- run: helm lint charts/openbot --values charts/openbot/ci/${{ matrix.target }}-values.yaml
# A key encryption key is a real 32 bytes rather than a placeholder, because the chart checks
# its shape. Generated here so no example key is ever a literal in this repository.
- name: Render
run: |
helm template ci charts/openbot \
--values charts/openbot/ci/${{ matrix.target }}-values.yaml \
--set-string secrets.keyEncryptionKey="$(openssl rand -base64 32)" \
--api-versions agents.x-k8s.io/v1beta1/Sandbox \
--api-versions extensions.agents.x-k8s.io/v1beta1/SandboxTemplate \
> rendered.yaml
# Rendering proves the templates run. This proves the result is coherent, which is a different
# question: every secret key a container demands has to be one the chart actually writes.
# Getting that wrong is invisible until a pod starts, and every shipped target had it wrong.
- run: bun scripts/check-rendered-chart.ts rendered.yaml
# And that the refusals are load-bearing rather than decorative. A chart full of `fail`
# messages nothing ever triggers is a chart that has never been shown to refuse anything, and
# every one of these describes a state that shipped in a values file at some point.
- name: Refusals fire
run: |
set -uo pipefail
refuses() {
local why="$1"; shift
if helm template ci charts/openbot \
--values charts/openbot/ci/${{ matrix.target }}-values.yaml \
--set-string secrets.keyEncryptionKey="$(openssl rand -base64 32)" \
--api-versions agents.x-k8s.io/v1beta1/Sandbox \
--api-versions extensions.agents.x-k8s.io/v1beta1/SandboxTemplate \
"$@" >/dev/null 2>&1; then
echo "::error::The chart rendered $why, which it is supposed to refuse."
return 1
fi
echo "refused: $why"
}
# The public example key, which the server will not start with. Only where this chart
# holds the secret: with a store, the value is not readable at template time, so the
# refusal is deliberately not armed and asserting it here would be asserting a bug.
if ! grep -qE '^ *enabled: true' <(sed -n '/^externalSecrets:/,/^[a-z]/p' charts/openbot/ci/${{ matrix.target }}-values.yaml); then
refuses "the public example encryption key" \
--set-string secrets.keyEncryptionKey="$(head -c 32 /dev/zero | base64)"
else
echo "skipped: the example-key refusal is not armed when the secret comes from a store"
fi
# A Bot endpoint with nothing on the request that says who is calling. Armed whether the
# secret is this chart's or a store's, because the key list is readable either way.
refuses "a managed agent URL with no token" \
--set-string config.managedAgent.url=http://agent.default:8000/ag-ui
# A browser inside every replica of a replicated API.
refuses "an embedded browser across several replicas" \
--set server.embeddedComputer=true --set server.replicaCount=2
# A Bot's egress proxy on a port the computer's own network policy does not allow. The
# variables reach the computer through extraEnv, so nothing else notices that the policy
# then refuses to let it be reached.
refuses "an egress proxy the network policy blocks" \
--set networkPolicy.enabled=true \
--set computers.extraEnv[0].name=EGRESS_PROXY_DEFAULT \
--set-string computers.extraEnv[0].value=http://proxy.internal:3128
test:
name: tests
runs-on: ubuntu-latest
# The suite includes a real database integration test. Without a database it fails on every run,
# including on main, which trains everyone to read a red CI as normal. pgvector rather than plain
# postgres because the knowledge schema uses the extension.
services:
postgres:
image: pgvector/pgvector:pg17
env:
POSTGRES_DB: openbot
POSTGRES_USER: openbot
POSTGRES_PASSWORD: openbot
ports:
- 5432:5432
options: >-
--health-cmd "pg_isready -U openbot -d openbot"
--health-interval 5s
--health-timeout 5s
--health-retries 10
env:
DATABASE_URL: postgres://openbot:openbot@localhost:5432/openbot
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
with:
bun-version: 1.3.14
- run: bun install --frozen-lockfile
# The Bots are not root workspaces and each keeps its own lockfile. The startup tests spawn
# the real entrypoint, so its dependencies have to be installed as well.
- run: bun install --frozen-lockfile
working-directory: agent-bot
# Same for the LangGraph Bot: its history tests import @langchain/core, which lives in that
# Bot's own tree and not in the root one.
- run: bun install --frozen-lockfile
working-directory: agent-langgraph
# Not the db:migrate script: that one loads ../.env, which does not exist in CI. DATABASE_URL
# comes from the job env instead, which drizzle.config.ts already reads.
- run: bunx drizzle-kit migrate --config=drizzle.config.ts
working-directory: server
# A passing job must include the expected test floor. Import-time failures can otherwise skip
# files before their tests are registered.
- run: bun run test:ci
build:
name: build
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
with:
bun-version: 1.3.14
- run: bun install --frozen-lockfile
- run: bun run build
# Migration files and the schema they were generated from, checked against each other. A snapshot
# that has drifted from the schema produces a migration nobody wrote, applied to somebody's
# database on their next deploy. Neither command needs a running database, only the config.
migrations:
name: migrations
runs-on: ubuntu-latest
env:
# drizzle.config.ts refuses to load without it. Nothing here connects.
DATABASE_URL: postgres://openbot:openbot@localhost:5432/openbot
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
with:
bun-version: 1.3.14
- run: bun install --frozen-lockfile
# Collisions and gaps between the migration files themselves.
- run: bunx drizzle-kit check --config=drizzle.config.ts
working-directory: server
# And the other direction: a schema change with no migration written for it. `generate` writes a
# file when it finds one, so the tree being dirty afterwards is the failure.
- name: Schema has no unwritten migration
working-directory: server
run: |
set -euo pipefail
bunx drizzle-kit generate --config=drizzle.config.ts --name=ci_drift_probe
if [ -n "$(git status --porcelain drizzle)" ]; then
echo "::error::The schema has changed without a migration. Run drizzle-kit generate and commit it."
git status --porcelain drizzle
exit 1
fi
# The image is the artefact people deploy, and almost nothing about whether it works is visible to
# the checks above. A dangling symlink, a supervised service that exits, a missing binary: all of
# them typecheck, lint and test perfectly.
image:
name: image
runs-on: ubuntu-latest
# The only slow job: it builds the container and boots it, ~14 minutes. The four checks above are
# seconds and stay on every push, because they catch most things cheaply. This one is the cost, so
# on a pull request it runs only when an admin adds the `full-ci` label — the point at which the PR
# is actually a merge candidate. On main and through the release workflow_call it always runs, so
# nothing reaches a release without it. `verify` treats a skipped job as passing, so an unlabelled
# PR is green on the cheap checks alone.
if: >-
github.event_name != 'pull_request' ||
contains(github.event.pull_request.labels.*.name, 'full-ci')
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4.3.0
# Loaded rather than pushed: this runs on pull requests, including from forks, and it proves
# the image builds without granting anything the ability to publish one.
- uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
with:
context: .
load: true
tags: openbot:ci
cache-from: type=gha
cache-to: type=gha,mode=max
# Building it says the layers resolve. Running it says the supervision tree comes up and stays
# up, which is a different claim and the one that has broken before.
- name: The image boots and serves
run: |
set -euo pipefail
# `OPENBOT_SINGLE_USER=true` because this boots a deployment with no identity provider, and
# the image sets NODE_ENV=production, where that combination refuses to start rather than
# serve an open deployment. Saying so is the point: the flag is how somebody declares they
# meant it. This was `OPENBOT_DEV_NO_AUTH=1` before, which the code never accepted at all,
# since it compares against the exact string "true"; it did nothing and nothing noticed.
#
# Placeholders, not secrets. `loadConfig` refuses to start without Intelligence and a
# licence configured, but it only checks that they are present and well-formed; nothing is
# contacted at start-up and /api/capabilities reads config alone. So this proves the image
# boots and serves without needing a licence, which is the part CI cannot have: a licence
# is bound to the machine it was issued for. Whether Intelligence actually answers is what
# the smoke journey checks, on a real deployment.
docker run -d --name openbot-ci -p 3001:3001 \
-e EMBEDDED_POSTGRES=on \
-e KEY_ENCRYPTION_KEY="$(openssl rand -base64 32)" \
-e TRUSTED_ORIGINS=http://localhost:3001 \
-e OPENBOT_SINGLE_USER=true \
-e INTELLIGENCE_API_URL=https://api.intelligence.copilotkit.ai \
-e INTELLIGENCE_GATEWAY_WS_URL=wss://realtime.intelligence.copilotkit.ai \
-e INTELLIGENCE_API_KEY=ci-not-a-real-key \
-e COPILOTKIT_LICENSE_TOKEN=ci-not-a-real-licence \
openbot:ci
for attempt in $(seq 1 150); do
if curl -fsS http://localhost:3001/api/capabilities >/dev/null 2>&1; then
echo "answered after ${attempt}s"
exit 0
fi
if [ -z "$(docker ps -q -f name=openbot-ci)" ]; then
echo "::error::The container exited before it answered."
docker logs openbot-ci
exit 1
fi
sleep 1
done
echo "::error::No answer on /api/capabilities within 150s."
docker logs openbot-ci
exit 1
- name: Supervision tree is stable, not respawning
run: |
set -euo pipefail
# A supervised service that exits is restarted forever. That looks healthy from outside for
# as long as something else is answering, so the log is where it shows.
sleep 15
if docker logs openbot-ci 2>&1 | grep -Eic 'restarting|respawn' | grep -qv '^0$'; then
echo "::error::A supervised service is restarting."
docker logs openbot-ci 2>&1 | grep -Ei 'restarting|respawn' | head -20
exit 1
fi
test -n "$(docker ps -q -f name=openbot-ci)" || {
echo "::error::The container is no longer running after 15s."
docker logs openbot-ci
exit 1
}
- if: always()
run: docker rm -f openbot-ci >/dev/null 2>&1 || true
# One check for branch protection to require. A new job above is covered by this without anybody
# remembering to add it to a list, and a job that was skipped for the wrong reason is not a pass.
verify:
name: verify
runs-on: ubuntu-latest
if: always()
needs: [static, deployables, chart, test, build, migrations, image]
steps:
- name: Require every check
env:
RESULTS: ${{ join(needs.*.result, ' ') }}
run: |
set -euo pipefail
echo "$RESULTS"
for result in $RESULTS; do
case "$result" in
success|skipped) ;;
*) echo "::error::A required check reported $result"; exit 1 ;;
esac
done