fix(export): unknown durations, stuck-progress watchdog, e2e export tests

.github/workflows/test.yml

@@ -0,0 +1,62 @@
+name: Tests
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+permissions:
+  contents: read
+
+concurrency:
+  group: tests-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  unit:
+    name: Unit tests (Vitest)
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    steps:
+      - uses: actions/checkout@v6
+      - uses: oven-sh/setup-bun@v2
+      - run: bun install --frozen-lockfile
+      - run: bun run test --run
+
+  e2e:
+    name: Export end-to-end (Playwright)
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+    steps:
+      - uses: actions/checkout@v6
+      - uses: oven-sh/setup-bun@v2
+      - run: bun install --frozen-lockfile
+
+      # ffmpeg makes the input fixtures and ffprobe checks the exported files.
+      - name: Install ffmpeg
+        run: sudo apt-get update && sudo apt-get install -y --no-install-recommends ffmpeg
+
+      - name: Make video fixtures
+        run: scripts/make-fixtures.sh --small
+
+      # Telemetry off: CI must never report to Datadog.
+      - name: Build the static site
+        run: bun run build
+        env:
+          NEXT_PUBLIC_DATADOG_ENV: "off"
+
+      - name: Install Chromium for Playwright
+        run: bunx playwright install --with-deps chromium
+
+      # Served with COOP/COEP (multi-threaded core, as in production) and
+      # without (single-threaded core); see playwright.config.ts.
+      - name: Run Playwright
+        run: bunx playwright test
+
+      - name: Upload Playwright report
+        if: failure()
+        uses: actions/upload-artifact@v4
+        with:
+          name: playwright-results
+          path: test-results/
+          retention-days: 7

.gitignore

@@ -50,3 +50,10 @@ chromatic.log
 
 # Local Netlify folder
 .netlify
+
+# generated video fixtures (scripts/make-fixtures.sh)
+/test/fixtures/
+
+# playwright
+/test-results/
+/playwright-report/

docs/DATADOG.md

@@ -44,8 +44,8 @@ its own, so we can see which browser and device combinations fail.
 |---|---|---|
 | `export_started` | Export clicked and settings validated | `format`, `resolution`, `input_size`, `clip_length`, `has_text`, `has_music`, `has_image_overlay`, `is_retry`, `retry_after` (retries only: the previous failure's category) |
 | `export_succeeded` | Terminal | the context above, plus `engine`, `fell_back` (true after an engine fallback), `durationMs` (click to done), `encode_ms`, `output_size` |
-| `export_failed` | Terminal | the context, plus `engine`, `durationMs`, `stage` (`engine` / `encode`), `category` |
-| `export_cancelled` | Terminal: the user cancelled | the context, plus `engine`, `durationMs`, `stage` |
+| `export_failed` | Terminal | the context, plus `engine`, `durationMs`, `stage` (`engine` / `encode`), `category`, and `stall_reason` for stalls |
+| `export_cancelled` | Terminal: the user cancelled | the context, plus `engine`, `durationMs`, `stage`, `stuck` (true when progress hadn't changed for 60 s: the user gave up on a hang, and monitors count it as a failure) |
 | `engine_fallback` | The multi-threaded engine stalled; the same export restarts automatically on the single-threaded engine. Not terminal: the attempt's outcome follows | the context, plus `from: multi`, `to: single`, `stall_reason` |
 | `ffmpeg_load_success` | A worker loaded the engine (once per worker) | `engine` (`multi` / `single`, what actually loaded), `engine_requested`, `single_reason` (`not_isolated` / `after_stall` / `mt_load_failed`, single-threaded only), `load_number` (`first` / `repeat`), `durationMs` |
 | `ffmpeg_load_failure` | A worker failed to load (not sent when the user cancelled) | `engine_requested`, `load_number`, `category` (`network` / `integrity` / `wasm` / `worker_script` / `unknown`), `durationMs` |
@@ -69,12 +69,21 @@ as part of the schema.
 | Category | Meaning |
 |---|---|
 | `engine_load` | The engine never became ready (any failure in the `engine` stage) |
-| `timeout` | No progress for 60 s; the next attempt uses the single-threaded engine |
+| `timeout` | The stall watchdog stopped the export (see `stall_reason`); the next attempt uses the single-threaded engine |
 | `unsupported_format` | FFmpeg couldn't decode the input |
 | `out_of_memory` | An allocation failed, or a `RangeError` |
 | `encoder_crash` | A WASM trap in the encoder (`memory access out of bounds`, `unreachable`) |
 | `network` | A network error after the engine started |
 | `encode_error` | Any other FFmpeg failure |
+| `unknown_duration` | The browser couldn't tell how long the clip is (`clip_length: unknown`), and the export then stalled or failed. Usually a live recording with no duration header that the seek-to-end fallback couldn't resolve |
+
+`export_failed.stall_reason` (stalls only):
+
+| Reason | Meaning |
+|---|---|
+| `no_activity` | No heartbeat or log line for 60 s: the engine is wedged (the multi-threaded thread-pool deadlock looked like this) |
+| `no_progress` | FFmpeg kept logging, but the encoded time didn't change for 120 s ("stuck at 100%") |
+| `overrun` | Encoding ran past 1.25 × the expected length + 15 s, for example looped music that never stops |
 
 ### Guarantees (covered by tests)
 
@@ -125,10 +134,25 @@ Created through the API on 2026-11-03 (site `datadoghq.com`):
 | Item | ID |
 |---|---|
 | Dashboard "Reframe — export reliability" | [`2tw-pge-qkk`](https://app.datadoghq.com/dashboard/2tw-pge-qkk) |
-| Monitor: exports failing in production (> 10 `export_failed` in 1 h; warning at 5) | `340137974` |
+| Monitor: exports not completing (started − succeeded − ordinary cancellations > 10 in 1 h; warning at 5). Counts failures, stuck cancellations (`stuck: true`) and abandoned exports | `340137974` |
 | Monitor: FFmpeg engine failing to load (> 5 `ffmpeg_load_failure` in 1 h) | `340138002` |
 | Monitor: cross-origin isolation missing (> 3 loads with `single_reason:not_isolated` in 1 h) | `340138004` |
 | Monitor: frontend errors spike (> 25 uncaught errors in 1 h) | `340138012` |
+| Monitor: exports start but none succeed (> 3 `export_started` and 0 `export_succeeded` in 6 h) | `340168191` |
+| RUM-based metrics `reframe.export.started` and `reframe.export.succeeded` (production action counts, from 2026-11-03) | — |
+| SLO: 95% of started exports succeed, 30 days (and 7 days), warning at 97% | `0ac26b24a9a156729eaa2bdab83d07ff` |
+| Monitor: SLO error budget burning > 6× over 6 h and 30 min | `340187598` |
+
+Why the "none succeed" and "not completing" monitors exist: during the
+multi-threaded engine outage (2026-10-22 to 2026-11-03) every export hung,
+and hung exports produced no `export_failed` at all, only cancellations or
+nothing. The original failure monitor counted only `export_failed`, so it
+stayed OK the whole time.
+
+The SLO counts every start that didn't end in success, so user
+cancellations count against it as well; that is deliberate (a cancel is
+usually an export that was too slow). With today's traffic a single failure
+moves it a lot; treat it as meaningful after a few hundred exports.
 
 The monitors only watch `env:production`, never page anyone, and email the
 maintainer. Their thresholds are absolute counts, chosen for today's low

docs/datadog/export-reliability.dashboard.json

@@ -1,5 +1,5 @@
 {
-  "title": "Reframe — export reliability",
+  "title": "Reframe \u2014 export reliability",
   "description": "Export outcomes from Reframe's browser RUM events (src/lib/exportTelemetry.ts). Success rate = export_succeeded / (export_succeeded + export_failed); cancellations are excluded. Treat groups with few finished exports as noise. Setup and facets: docs/DATADOG.md.",
   "layout_type": "ordered",
   "template_variables": [
@@ -1009,6 +1009,56 @@
           }
         ]
       }
+    },
+    {
+      "definition": {
+        "title": "Stalled exports by reason and engine",
+        "type": "query_table",
+        "requests": [
+          {
+            "response_format": "scalar",
+            "queries": [
+              {
+                "data_source": "rum",
+                "name": "stalled",
+                "indexes": [
+                  "*"
+                ],
+                "compute": {
+                  "aggregation": "count"
+                },
+                "search": {
+                  "query": "@type:action service:reframe $env @action.target.name:export_failed @context.category:(timeout OR unknown_duration)"
+                },
+                "group_by": [
+                  {
+                    "facet": "@context.stall_reason",
+                    "limit": 3,
+                    "sort": {
+                      "aggregation": "count",
+                      "order": "desc"
+                    }
+                  },
+                  {
+                    "facet": "@context.engine",
+                    "limit": 3,
+                    "sort": {
+                      "aggregation": "count",
+                      "order": "desc"
+                    }
+                  }
+                ]
+              }
+            ],
+            "formulas": [
+              {
+                "formula": "stalled",
+                "alias": "Stalled exports"
+              }
+            ]
+          }
+        ]
+      }
     }
   ]
 }

package.json

@@ -31,6 +31,7 @@
     "wasm-feature-detect": "^1.8.0"
   },
   "devDependencies": {
+    "@playwright/test": "1.64.1",
     "@storybook/addon-a11y": "^10.5.10",
     "@storybook/addon-docs": "^10.5.10",
     "@storybook/addon-themes": "^10.5.10",

playwright.config.ts

@@ -0,0 +1,44 @@
+import { defineConfig, devices } from "@playwright/test";
+
+// End-to-end export tests against the production build (out/), served with
+// the same cross-origin isolation headers as Netlify.
+//
+//   scripts/make-fixtures.sh --small
+//   NEXT_PUBLIC_DATADOG_ENV=off bun run build
+//   bunx playwright test
+//
+// Two projects: "multi-threaded" (COOP/COEP set, so @ffmpeg/core-mt loads,
+// as on the live site) and "single-threaded" (no isolation headers).
+export default defineConfig({
+  testDir: "test/e2e",
+  // Each test runs real FFmpeg.wasm encodes; one worker keeps CPU contention
+  // from making timings meaningless.
+  workers: 1,
+  fullyParallel: false,
+  timeout: 240_000,
+  expect: { timeout: 15_000 },
+  retries: process.env.CI ? 1 : 0,
+  reporter: process.env.CI ? [["list"], ["junit", { outputFile: "test-results/e2e-junit.xml" }]] : "list",
+  use: {
+    ...devices["Desktop Chrome"],
+    acceptDownloads: true,
+    trace: "retain-on-failure",
+  },
+  projects: [
+    { name: "multi-threaded", use: { baseURL: "http://127.0.0.1:4173" } },
+    { name: "single-threaded", use: { baseURL: "http://127.0.0.1:4174" } },
+  ],
+  webServer: [
+    {
+      command: "node scripts/serve-isolated.mjs 4173",
+      url: "http://127.0.0.1:4173",
+      reuseExistingServer: !process.env.CI,
+    },
+    {
+      command: "node scripts/serve-isolated.mjs 4174",
+      url: "http://127.0.0.1:4174",
+      env: { ISOLATED: "0" },
+      reuseExistingServer: !process.env.CI,
+    },
+  ],
+});

scripts/make-fixtures.sh

@@ -0,0 +1,60 @@
+#!/usr/bin/env bash
+# Recreates the video fixtures used by the end-to-end export tests.
+# Output goes to test/fixtures/ (gitignored). Needs the ffmpeg CLI
+# (macOS: `brew install ffmpeg`, Ubuntu: `sudo apt-get install ffmpeg`).
+#
+#   scripts/make-fixtures.sh          # all fixtures
+#   scripts/make-fixtures.sh --small  # everything except the ~300 MB clip
+set -euo pipefail
+
+cd "$(dirname "$0")/.."
+OUT=test/fixtures
+mkdir -p "$OUT"
+
+command -v ffmpeg >/dev/null || { echo "ffmpeg not found; install it first." >&2; exit 1; }
+
+ff() { ffmpeg -hide_banner -loglevel error -y "$@"; }
+
+# A moving test pattern with a tone, so trims, speed changes and audio edits
+# are visible in the output.
+video() { echo "testsrc2=size=$1:rate=30:duration=$2"; }
+tone() { echo "sine=frequency=440:sample_rate=48000:duration=$1"; }
+
+echo "normal.mp4: 1280x720 H.264/AAC, 10 s"
+ff -f lavfi -i "$(video 1280x720 10)" -f lavfi -i "$(tone 10)" \
+  -c:v libx264 -pix_fmt yuv420p -preset veryfast -c:a aac -b:a 128k \
+  -movflags +faststart -shortest "$OUT/normal.mp4"
+
+# Browser screen recordings (MediaRecorder) are written as a live stream: the
+# WebM header has no duration and there are no cues, so <video>.duration is
+# Infinity until the whole file is read. `-live 1` produces the same layout.
+echo "no-duration.webm: 1280x720 VP8/Opus, 6 s, no duration in the container"
+ff -f lavfi -i "$(video 1280x720 6)" -f lavfi -i "$(tone 6)" \
+  -c:v libvpx -b:v 1M -deadline realtime -cpu-used 8 -c:a libopus \
+  -f webm -live 1 -shortest "$OUT/no-duration.webm"
+
+echo "portrait.mp4: 720x1280 H.264/AAC, 6 s"
+ff -f lavfi -i "$(video 720x1280 6)" -f lavfi -i "$(tone 6)" \
+  -c:v libx264 -pix_fmt yuv420p -preset veryfast -c:a aac -b:a 128k \
+  -movflags +faststart -shortest "$OUT/portrait.mp4"
+
+echo "no-audio.mp4: 1280x720 H.264, 6 s, no audio track"
+ff -f lavfi -i "$(video 1280x720 6)" \
+  -c:v libx264 -pix_fmt yuv420p -preset veryfast -an \
+  -movflags +faststart "$OUT/no-audio.mp4"
+
+if [[ "${1:-}" != "--small" ]]; then
+  # Noise defeats compression, so a constant 40 Mb/s for 60 s lands near 300 MB.
+  echo "large.mp4: 1920x1080 H.264/AAC, 60 s, ~300 MB (memory testing)"
+  ff -f lavfi -i "$(video 1920x1080 60),noise=alls=60:allf=t" -f lavfi -i "$(tone 60)" \
+    -c:v libx264 -pix_fmt yuv420p -preset ultrafast \
+    -b:v 40M -minrate 40M -maxrate 40M -bufsize 80M \
+    -c:a aac -b:a 128k -movflags +faststart -shortest "$OUT/large.mp4"
+fi
+
+echo
+for f in "$OUT"/*.{mp4,webm}; do
+  [[ -e "$f" ]] || continue
+  printf '%-28s %8s  duration=%s\n' "$f" "$(du -h "$f" | cut -f1)" \
+    "$(ffprobe -v error -show_entries format=duration -of csv=p=0 "$f")"
+done