Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ site/src/assets/landing/*.mp4
# Local browser-test outputs.
ui/host/playwright-report/
ui/host/test-results/
ui/host/e2e/__screenshots__/
ui/host/tsconfig.tsbuildinfo

# Docs
Expand Down
37 changes: 35 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -125,11 +125,11 @@ jobs:
- group: smoke and accessibility
slug: smoke
skip_performance: "1"
args: ""
args: --project=app-1100-light --project=app-1100-dark --project=app-1440-light --project=app-1440-dark --project=mcp
- group: performance
slug: performance
skip_performance: ""
args: e2e/performance.spec.ts
args: e2e/specs/app/performance.spec.ts
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
Expand All @@ -146,6 +146,9 @@ jobs:
node-version: 24.19.0
- name: just ui-deps
run: just ui-deps
- name: just e2e-browsers
if: matrix.slug == 'smoke'
run: just e2e-browsers
- name: just e2e
run: just e2e ${{ matrix.args }}
env:
Expand All @@ -161,6 +164,36 @@ jobs:
if-no-files-found: ignore
retention-days: 7

visual:
name: Browser visual
runs-on: ubuntu-24.04
timeout-minutes: 10
container:
image: mcr.microsoft.com/playwright:v1.64.1-noble@sha256:7852dc5a7d72f5fe458a597d690f6c42fbc8e9e26ddfa95f8e6ef552558d9bbf
options: --ipc=host
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
# The browser image lacks unzip, which setup-bun needs.
- name: Install Bun
run: npm install --global bun@1.4.2
- name: Install UI dependencies
run: cd ui/host && bun install --frozen-lockfile
- name: Check image driver version
run: node scripts/playwright-image-check.mjs
- name: Visual comparisons
run: cd ui/host && bun run e2e --project visual
- name: Upload browser evidence and Linux baselines
if: always()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: browser-evidence-visual
path: |
ui/host/playwright-report/
ui/host/test-results/
ui/host/e2e/__screenshots__/visual/linux/
if-no-files-found: ignore
retention-days: 7

# Build the exact runtime image on pull requests and reject high/critical OS,
# Go-binary, or embedded-secret findings before merge.
container-scan:
Expand Down
121 changes: 112 additions & 9 deletions docs/testing.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,112 @@
# Browser smoke tests
# Browser tests

Install Go, Bun, just and Google Chrome, then run `just ui-deps` and
`just e2e-browsers`. The latter invokes the locked Playwright CLI to install its
bundled Chromium and required Linux OS libraries; CI smoke uses the same recipe
before running MCP tests on a clean Ubuntu runner. Linux may request sudo for OS
packages. From `ui/host`, run `bun run e2e --help` for the pinned Playwright CLI.
The app matrix uses system Google Chrome. No AI provider key is
needed. Run `just e2e` from the repository root to rebuild the embedded SPA,
MCP App and `bin/fanout`, then run every surface. `just check` remains browser-free.

One config owns `ui/host/e2e/specs/`: `app/` contains the existing seeded-server
specs, `mcp/` checks the built MCP App through the official AppBridge, and
`visual/` contains screenshot assertions. Each spec imports the composed
`e2e/fixtures/test` runner; `just ui-boundaries-check` enforces this, and its
failure-injection tests run in `just script-tests` and `just check`.
`fixtures/` separates app sessions, browser faults, mock-host navigation,
accessibility scans and keyboard navigation. `data/` contains synthetic JSON;
`support/` contains shared helpers and their unit tests. Vitest still discovers
all `*.test.ts` there and excludes Playwright specs. Site recording helper tests
still run through the site's existing test commands.

The app projects are `app-1100-light`, `app-1100-dark`, `app-1440-light` and
`app-1440-dark`, each with one worker. Their shared `setup` dependency starts the
seeded backend once; global teardown stops it after dependent projects finish.
Rerunning setup in UI Mode stops owned instances before replacement. Each launch
has a unique identity, PID and private directory in a separate ownership registry;
readiness must match the spawned PID and identity. Teardown checks ownership before
signalling and removes every registered instance and its session/data directory,
including failed launches, even if shared readiness was overwritten.
Run ownership regressions without a browser using
`bun test e2e/support/app-lifecycle.test.ts` from `ui/host`;
`bun run test -- e2e/support` runs all support tests with the configured Vitest
environment.
MCP and visual projects have no setup dependency and never boot Fanout.

With Go, Bun, just and **Google Chrome** installed, run `just e2e` from the
repository root. It rebuilds the embedded SPA and MCP app plus `bin/fanout`,
then runs the pinned Playwright suite in `ui/host/e2e/` using system Chrome;
no browser download or AI provider key is needed. `just check` remains browser-free.
```sh
just e2e-browsers # locked Chromium and its OS dependencies
just e2e # every surface, including performance
just e2e-mcp # recorded MCP App protocol and accessibility
just e2e-visual # screenshot comparisons on this platform
just e2e-ui # Playwright UI Mode
just e2e-update-snapshots # accept local visual changes after review
just e2e-linux-baselines # accept Linux visuals in the pinned CI image
just e2e e2e/specs/app/performance.spec.ts
```

Screenshot baselines live under
`ui/host/e2e/__screenshots__/<project>/<platform>/<spec>/<name>.png`. Review
actual/diff images before accepting changes; update only the visual project.
Darwin baselines are generated locally. Linux baselines must come from the pinned
Playwright image used by `just e2e-linux-baselines` and CI, with `--ipc=host`.
The recipe mounts this checkout and installs Bun through npm because the image
lacks unzip; locked UI dependencies use a separate container volume.
In UI Mode, enable and run the `setup` project before app tests.
No Go build is needed for visual tests. `just e2e-image-check`
fails if the image's `/ms-playwright/.docker-info` driver version differs from
installed `@playwright/test`. CI uploads reports, results and any generated Linux
baselines. Missing Linux baselines fail the comparison and are available in the
artifact for review and acceptance; CI never silently accepts changed pixels.

The existing `app/screenshots.spec.ts` stays in app: it captures opt-in review
images and manifests, but has no screenshot assertions. Its
`FANOUT_E2E_SCREENSHOTS` gate is unchanged.

## Recorded MCP host

`e2e/host/server.mjs` builds current `panels.html` with `vite.apps.config.ts`,
setting `FANOUT_APPS_OUT` to a temporary directory, then builds the test host.
It serves only those production files on `127.0.0.1` port zero with
`Cache-Control: no-store`; Playwright captures `listening on <port>` into
`E2E_PORT`. SIGTERM closes the listener and removes build output. It never
changes the embedded HTML and adds no product route.

The mock host selects scenario/theme via query parameters and embeds the view
in an `allow-scripts` sandbox. AppBridge sends tool input and structured results
with a JSON text mirror, or cancellation, after initialization. A held result
checks loading; an unbridged scenario checks the SDK connection deadline with
Playwright's clock. Host context carries light/dark theme and inline display
mode. Recorded helper calls are matched by complete input; unrecorded calls,
resource reads, links and other requests fail loudly. The view currently requests
only server tools and size changes; the host can read the built panels resource.

To refresh recordings, build the server and run from the repository root:

```sh
just build
cd ui/host
bun e2e/host/record.ts
```

The recorder starts the same disposable `e2eseed` instance as app tests, calls
real MCP tools over its authenticated loopback connection, saves only synthetic
inputs/results to `e2e/data/mcp.json`, and cleans up. It rejects external-instance
configuration. It records overview health, topology, performance/Slow operations,
trace detail/correlated logs, log search, a custom panel, no-data guidance and a
tool error, plus refresh, variable, exemplar and trace helper results. Absolute recorded
windows keep assertions independent of the current clock. Credentials remain in
private temporary state and are never recorded.

MCP specs cover both themes, loading, every recorded view, no-data, errors,
cancellation, connection failure, serious/critical axe violations and keyboard
access to controls. Visual specs cover overview, performance and trace in both
themes. These tests do not cover real ChatGPT/Claude host integration, remote
MCP/OAuth handshakes, live telemetry, AI answers, arbitrary helper input, mobile
layouts or every visual state. App specs retain real-server and handcrafted
fragment coverage below.

## Seeded backend lifecycle

Setup creates a disposable loopback server, first administrator and near-now
deterministic OTLP protobuf traces and logs. Teardown stops the server and removes
Expand All @@ -28,10 +131,10 @@ tooltip text without reading canvas pixels. Fragment height is capped at 2000
pixels to match the host, and messages must stop for five seconds after the first
settled frame. CI pins Node 24.19.0 and limits the job to 15 minutes. It runs the same recipe on
pull requests and pushes. Failures write `ui/host/playwright-report/index.html`
and retain traces/screenshots in `ui/host/test-results/`; both directories are
and retain failure screenshots and first-retry traces in `ui/host/test-results/`; both directories are
ignored by git. CI uploads available browser evidence after every run.

For browser-free verification, run `bunx playwright test --list` in `ui/host`.
For browser-free discovery, run `bun run e2e --list` in `ui/host`.
`bun run test` runs unit tests and excludes Playwright specs.

## Dashboard performance measurements
Expand Down Expand Up @@ -118,7 +221,7 @@ Assess WCAG AA text contrast and chart marks against their actual surfaces.
Check all six ordered categorical colours plus Other under colour-vision
deficiency simulation. Browser accessibility scans cannot read canvas text, so
supplement any scan with compiler/token assertions and screenshots.
CI runs the performance spec (`just e2e e2e/performance.spec.ts`) in its own job and
CI runs the performance spec (`just e2e e2e/specs/app/performance.spec.ts`) in its own job and
the rest with `FANOUT_E2E_SKIP_PERFORMANCE=1`. `just e2e` runs the axe-core AA scan of dashboards and chat panels in both
themes; `FANOUT_E2E_SCREENSHOTS=1 just e2e` also captures every panel type in
both themes and states for review under `ui/host/test-results/`. Record the
Expand Down Expand Up @@ -177,4 +280,4 @@ outside the verification environment's scope. Neither command publishes the site

Refresh the README's light and dark System overview captures with `just landing-captures` (real seeded server and local Chrome). To capture an existing instance, set `FANOUT_LANDING_BASE_URL` to its URL and `FANOUT_LANDING_SESSION_COOKIE` to a `name=value` session cookie before running the same command; no server starts and no data is seeded.

To record chat and dashboard clips, set `FANOUT_LANDING_RECORDINGS=1` and the external-instance variables above, then run `bun run e2e landing-recordings.spec.ts --project=1440-light --project=1440-dark` from `ui/host`. Follow [landing-recordings.md](../site/scripts/landing-recordings.md) to encode and upload them.
To record chat and dashboard clips, set `FANOUT_LANDING_RECORDINGS=1` and the external-instance variables above, then run `bun run e2e e2e/specs/app/landing-recordings.spec.ts --project=app-1440-light --project=app-1440-dark` from `ui/host`. Follow [landing-recordings.md](../site/scripts/landing-recordings.md) to encode and upload them.
6 changes: 3 additions & 3 deletions internal/cmd/e2eseed/main_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ func TestSeedPayloadsAreDeterministic(t *testing.T) {
}

func TestAllPanelsFixtureValidates(t *testing.T) {
data, err := os.ReadFile("../../../ui/host/e2e/fixtures/all-panels.json")
data, err := os.ReadFile("../../../ui/host/e2e/data/all-panels.json")
if err != nil {
t.Fatal(err)
}
Expand Down Expand Up @@ -105,7 +105,7 @@ func decodeFixture(data []byte, spec *panel.Dashboard) error {
}

func TestAllPanelsFixtureRejectsUnknownFields(t *testing.T) {
data, err := os.ReadFile("../../../ui/host/e2e/fixtures/all-panels.json")
data, err := os.ReadFile("../../../ui/host/e2e/data/all-panels.json")
if err != nil {
t.Fatal(err)
}
Expand Down Expand Up @@ -385,7 +385,7 @@ func (s *fixtureSubmitter) Submit(ctx context.Context, batch telemetrystore.Batc
}

func TestPerformanceFixtureValidatesAndExecutes(t *testing.T) {
data, err := os.ReadFile("../../../ui/host/e2e/fixtures/performance.json")
data, err := os.ReadFile("../../../ui/host/e2e/data/performance.json")
if err != nil {
t.Fatal(err)
}
Expand Down
2 changes: 1 addition & 1 deletion internal/mcp/apps/panels.html

Large diffs are not rendered by default.

33 changes: 31 additions & 2 deletions justfile
Original file line number Diff line number Diff line change
Expand Up @@ -64,15 +64,44 @@ ui: ui-lint ui-host ui-apps
build VERSION=`git describe --tags --always --dirty 2>/dev/null || echo dev`: ui
bash scripts/with-duckdb.sh go build -ldflags "-s -w -X main.version={{VERSION}}" -o bin/fanout ./cmd/fanout

# Install Chromium from the locked Playwright package, plus Linux OS libraries.
e2e-browsers:
cd ui/host && bun node_modules/playwright/cli.js install --with-deps chromium

# Disposable seeded instance and system Google Chrome; separate from check.
# Extra arguments go to Playwright, e.g. `just e2e e2e/performance.spec.ts`.
# Extra arguments go to Playwright, e.g. `just e2e e2e/specs/app/performance.spec.ts`.
e2e *ARGS:
just build
cd ui/host && bun run e2e {{ARGS}}

# Recorded MCP App protocol and accessibility checks; no Go server.
e2e-mcp *ARGS:
cd ui/host && bun run e2e --project mcp {{ARGS}}

# Screenshot comparisons against the current platform's accepted baselines.
e2e-visual *ARGS:
cd ui/host && bun run e2e --project visual {{ARGS}}

# Explore all surfaces with Playwright UI Mode.
e2e-ui *ARGS:
just build
cd ui/host && bun run e2e --ui {{ARGS}}

# Deliberately refresh only the visual surface.
e2e-update-snapshots *ARGS:
just e2e-visual --update-snapshots {{ARGS}}

# Check the browser image against the installed driver before taking pixels.
e2e-image-check:
node scripts/playwright-image-check.mjs

# Generate Linux baselines with the same image and shared-memory setting as CI.
e2e-linux-baselines:
docker run --rm --ipc=host -v "{{justfile_directory()}}:/work" -v /work/ui/host/node_modules -w /work mcr.microsoft.com/playwright:v1.64.1-noble@sha256:7852dc5a7d72f5fe458a597d690f6c42fbc8e9e26ddfa95f8e6ef552558d9bbf bash -euc 'npm install --global bun@1.4.2; cd ui/host; bun install --frozen-lockfile; cd ../..; node scripts/playwright-image-check.mjs; cd ui/host; bun run e2e --project visual --update-snapshots'

# Refresh the README's light and dark System overview captures.
landing-captures:
FANOUT_LANDING_BASE_URL="${FANOUT_LANDING_BASE_URL:-}" FANOUT_LANDING_SESSION_COOKIE="${FANOUT_LANDING_SESSION_COOKIE:-}" FANOUT_LANDING_CAPTURES=1 just e2e e2e/landing-captures.spec.ts
FANOUT_LANDING_BASE_URL="${FANOUT_LANDING_BASE_URL:-}" FANOUT_LANDING_SESSION_COOKIE="${FANOUT_LANDING_SESSION_COOKIE:-}" FANOUT_LANDING_CAPTURES=1 just e2e e2e/specs/app/landing-captures.spec.ts

# CI publishes ghcr.io/labstack/fanout; this is for trying the image locally
# without pushing anything.
Expand Down
5 changes: 5 additions & 0 deletions scripts/playwright-image-check.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
import { readFileSync } from "node:fs";
const image = JSON.parse(readFileSync("/ms-playwright/.docker-info", "utf8"));
const driver = JSON.parse(readFileSync(new URL("../ui/host/node_modules/@playwright/test/package.json", import.meta.url), "utf8"));
if (image.driverVersion !== driver.version) throw new Error(`Playwright image driver ${image.driverVersion} differs from installed ${driver.version}`);
console.log(`Playwright image and driver match: ${driver.version}`);
9 changes: 7 additions & 2 deletions scripts/ui-boundaries.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,12 @@ function workspace(file) {
return first === "host" ? "host" : "shared";
}

function check(file, specifier, offset, source) {
function check(file, specifier, offset, source, bindings = []) {
const spec = relative(ui, file).startsWith(`host${sep}e2e${sep}specs${sep}`) && /\.spec\.[jt]sx?$/.test(file);
if (spec && (specifier.startsWith("@playwright/test") || bindings.some(name => name === "test" || name === "expect") && resolve(dirname(file), specifier).replace(/\.ts$/, "") !== resolve(ui, "host/e2e/fixtures/test"))) {
console.error(`${relative(root, file)}:${source.slice(0, offset).split("\n").length}: specs must import test/expect from e2e/fixtures/test`);
failures++;
}
const owner = workspace(file);
const local = specifier.startsWith("./") || specifier.startsWith("../");
const target = local ? resolve(dirname(file), specifier) : null;
Expand All @@ -42,7 +47,7 @@ function scan(file) {
const literal = node.source ?? (node.type === "TSImportType" ? node.argument :
node.type === "TSExternalModuleReference" ? node.expression :
node.type === "CallExpression" && node.callee?.name === "require" ? node.arguments[0] : null);
if (literal?.type === "StringLiteral") check(file, literal.value, literal.start, source);
if (literal?.type === "StringLiteral") check(file, literal.value, literal.start, source, node.specifiers?.map(binding => binding.imported?.name ?? binding.exported?.name) ?? []);
else if (literal?.type === "TemplateLiteral" && literal.expressions.length === 0) {
check(file, literal.quasis[0].value.cooked, literal.start, source);
}
Expand Down
18 changes: 18 additions & 0 deletions scripts/ui-boundaries.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -50,3 +50,21 @@ test("allows workspace packages and package-free shared imports; skips comments
"host/node_modules/test/index.ts": 'import { X } from "../../../outside/src/x";',
}).exitCode).toBe(0);
});

test.each([
'import { test, expect } from "@playwright/test";',
'import { test as base } from "@playwright/test";',
'export { expect } from "@playwright/test";',
'const runner = import("@playwright/test");',
'import { test } from "../../support/accessibility-support";',
])("rejects bypassing composed Playwright fixtures: %s", source => {
const result = check({ "host/e2e/specs/mcp/panels.spec.ts": source });
expect(result.exitCode).toBe(1);
expect(result.stderr.toString()).toContain("specs must import test/expect from e2e/fixtures/test");
});

test("allows composed Playwright fixtures in every surface", () => {
expect(check(Object.fromEntries(["app", "mcp", "visual"].map(surface => [
`host/e2e/specs/${surface}/panels.spec.ts`, 'import { test, expect } from "../../fixtures/test";',
]))).exitCode).toBe(0);
});
2 changes: 1 addition & 1 deletion site/scripts/landing-recording-scroll.test.mjs
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { runInNewContext } from "node:vm";
import { revealRecording, waitForConversation } from "../../ui/host/e2e/landing-recording-support.ts";
import { revealRecording, waitForConversation } from "../../ui/host/e2e/support/landing-recording-support.ts";

// Execute the actual page.evaluate callbacks with a virtual layout and clock.
// No browser is launched; iframe growth is represented by moving the answer.
Expand Down
2 changes: 1 addition & 1 deletion site/scripts/landing-recordings.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ instance URL, and `FANOUT_LANDING_SESSION_COOKIE` to a `name=value` session cook
in the process environment. Run from `ui/host`:

```sh
bun run e2e landing-recordings.spec.ts --project=1440-light --project=1440-dark
bun run e2e e2e/specs/app/landing-recordings.spec.ts --project=app-1440-light --project=app-1440-dark
```

Recordings use a 1280 × 800 viewport and raw video at the same size; Playwright
Expand Down
Loading
Loading