ci(repo): test Expo releases before they publish

.changeset/expo-e2e-release-gate.md

@@ -0,0 +1,2 @@
+---
+---

.claude/skills/verify-clerk-expo/SKILL.md

@@ -84,7 +84,7 @@ The `backend` line says which backend the CLI chose and why. This transcript is
 
 A remote session has no Metro and no watch build. A worktree's iOS session and its Android session share the worktree's application, so their runs go one after the other too, and the second fails with `DEVICE_BUSY` while the first is driving. After an edit to the app or to a package, commit, push, and `run` again, and the same session builds the new commit and installs it. Specs run from your working tree, so an edit to a spec needs no commit. The session runs on a free GitHub-hosted runner, where `up` takes tens of minutes, nearly all of it the build. `down` stops the session at once, so run `down` as soon as you are done. [Remote devices](references/remote.md) has what the machine needs, the faster runner labels, the session limits, and what the session builds.
 
-On a Linux machine that can run the emulator, the CLI picks the local backend for Android and builds the test app on that machine, as the `Verify end-to-end tests` workflow does on its Linux runner.
+On a Linux machine that can run the emulator, the CLI picks the local backend for Android and builds the test app on that machine, as the `Expo E2E` workflow does on its Linux runner.
 
 ## Doctor
 
@@ -159,7 +159,7 @@ A spec file that needs other settings than the standard ones has a settings file
 
 ### Check your work
 
-The specs under `specs/features/<feature>/` are committed and cover the feature map in `features/`, which starts at `features/README.md`. Prove your own change, on both platforms when the change is not specific to one, and leave the rest of the committed specs to PR CI (`.github/workflows/verify-e2e.yml`), which runs every one of them once the pull request is ready for review. When your change is to behavior a committed spec already covers, that spec is your proof: run it. A spec covers a behavior only when it asserts on it. A spec that passes through the screen you changed without checking what you changed is not your proof. Run another feature's specs yourself only when you changed code that feature shares and want to know before CI does. For new work:
+The specs under `specs/features/<feature>/` are committed and cover the feature map in `features/`, which starts at `features/README.md`. Prove your own change, on both platforms when the change is not specific to one. CI runs every committed spec only on the Version packages pull request (`changeset-release/main`), before each release, and never on your pull request (`.github/workflows/verify-e2e.yml`). When your change is to behavior a committed spec already covers, that spec is your proof: run it. A spec covers a behavior only when it asserts on it. A spec that passes through the screen you changed without checking what you changed is not your proof. Run another feature's specs too when you changed code that feature shares. For new work:
 
 1. Write a spec under `specs/explored/`, which is gitignored. It imports the fixture as `'../fixtures.ts'`.
 2. Run it by path.
@@ -319,7 +319,7 @@ For a remote device, `down` ends the runner job and waits for it to finish, and
 - `src/host.ts`, `src/fixture.ts`, `src/native-build.ts`, and `src/freshness.ts` are this repository's own: the build of the test app, the native build that CI keeps between runs, the Metro ports, and the check that Metro serves current JS. `specs/app.ts` names the test app and its entry for a dev client, and `specs/native.ts` holds the per-platform locators for the native views and the locators of the home's links. Both are this repository's own too.
 - `npm test --prefix integration/expo-native` runs the CLI's unit tests, with no network, key, or device. `npm run typecheck --prefix integration/expo-native` runs `tsc`. The `Expo Native Runner Tests` job in `.github/workflows/ci.yml` runs both on Linux when a pull request changes the package, the test app, or a package the test app links.
 - `run --github-report` hands the results of the run to `@e2e-dev/github` as one report. The reporter writes the report to the job summary. With a `GITHUB_TOKEN` that may write pull request comments, it also posts one comment on the pull request and updates that comment on later runs. The reporter never changes the exit code, and nothing is reported for a run with a tainted file.
-- `.github/workflows/verify-e2e.yml`, the `Verify end-to-end tests` workflow, runs `up --backend local`, `run --all --retries 1 --max-failures 5 --github-report`, and `down` on a runner for each platform, with a device and a Clerk application for each. It starts on a pull request to `main` that changes the package, the test app, or one of the four packages the test app links. It runs for a pull request that is not a draft, and a draft or a pull request from a fork gets a notice instead. Start it by hand with `gh workflow run verify-e2e.yml --ref <branch>`. A failing spec shows on the pull request and is not required for a merge, and a test that fails and then passes on its one retry is `flaky` and does not fail the job. The runners are GitHub-hosted, `macos-26` and `ubuntu-24.04`, and the repository variables `VERIFY_CI_RUNNER_IOS` and `VERIFY_CI_RUNNER_ANDROID` name other labels. The Platform API key comes from the `MOBILE_VERIFICATION_PLATFORM_API_KEY` repository secret. The workflow sets `VERIFY_LOCAL_BUILD=standalone` and `VERIFY_NATIVE_CACHE` ([freshness.md](references/freshness.md)). It uploads each run's `run.json`, app log, driver logs, video, screenshots, and e2e's `report.json`, `junit.xml`, summary, and failure pages for three days, and only when no secret is found in the run. `bin/configure-ios-simulators.sh` speeds up animations and turns off autocorrect and the keyboard tutorials on each simulator in `CLERK_TEST_DEVICES`, and this workflow and `.github/workflows/verify-remote.yml` call it.
+- `.github/workflows/verify-e2e.yml`, the `Expo E2E` workflow, runs `up --backend local`, `run --all --retries 1 --max-failures 5 --github-report`, and `down` on a runner for each platform, with a device and a Clerk application for each. It runs on the Version packages pull request (`changeset-release/main`), so each release is tested before it publishes, and the `Expo E2E` check fails when a spec fails. On other pull requests the device tests skip and the `Expo E2E` check passes. Start it by hand with `gh workflow run verify-e2e.yml --ref <branch>`. A test that fails and then passes on its one retry is `flaky` and does not fail the job. The runners are Blacksmith's `blacksmith-6vcpu-macos-26` and `blacksmith-8vcpu-ubuntu-2204`, and the repository variables `VERIFY_CI_RUNNER_IOS` and `VERIFY_CI_RUNNER_ANDROID` name other labels. The Platform API key comes from the `MOBILE_VERIFICATION_PLATFORM_API_KEY` repository secret. The workflow sets `VERIFY_LOCAL_BUILD=standalone` and `VERIFY_NATIVE_CACHE` ([freshness.md](references/freshness.md)). It uploads each run's `run.json`, app log, driver logs, video, screenshots, and e2e's `report.json`, `junit.xml`, summary, and failure pages for three days, and only when no secret is found in the run. `bin/configure-ios-simulators.sh` speeds up animations and turns off autocorrect and the keyboard tutorials on each simulator in `CLERK_TEST_DEVICES`, and this workflow and `.github/workflows/verify-remote.yml` call it.
 - The `Expo` workflow, `.github/workflows/expo-native-build.yml`, runs nothing on a device. It builds the test app as a Release app on Expo SDK 55 and 58 for each platform, and runs the Android unit tests of `@clerk/expo-biometrics`.
 - `.github/workflows/verify-attach.yml` publishes the evidence that `attach` hands to a session's runner, with `scripts/verify-attach.mjs`. It reads the manifest that `src/core/remote/handoff.ts` of `@clerk/internal-mobile-e2e` writes. It and the package's `src/core/publish.ts` each replace the block the other wrote, each applies the same rules to the comments and to the start of the line, and each lays the files out in the same tables. The script is checked out alone, so it imports nothing from the package. So a change to the manifest, the comments, those rules, or the tables is a change in both places. `scripts/verify-attach.test.mjs` has its tests.
 - A remote session runs in `.github/workflows/verify-remote.yml`. `src/platform/session-device.ts` holds the steps that the session runs on its device, and `src/core/remote/` of `@clerk/internal-mobile-e2e` is the code on both ends of it. The runner installs the package with `npm ci`. A session keeps the core it installed when it started, so after a change to the commit pin of `@clerk/internal-mobile-e2e`, commit, push, `down`, then `up`.

.claude/skills/verify-clerk-expo/references/freshness.md

@@ -35,7 +35,7 @@ The check dates a Metro revision it has not seen before by the second it was bui
 
 ## A remote lease has none of this
 
-A remote lease runs a standalone Release build with the JS embedded, so there is no watch build, no Metro, and none of the three checks. Its build key also covers the JS inputs, and a JS edit reaches the app by commit, push, `run`. See [remote.md](remote.md). The `Verify end-to-end tests` workflow sets `VERIFY_LOCAL_BUILD=standalone`, so its local devices run the same kind of app.
+A remote lease runs a standalone Release build with the JS embedded, so there is no watch build, no Metro, and none of the three checks. Its build key also covers the JS inputs, and a JS edit reaches the app by commit, push, `run`. See [remote.md](remote.md). The `Expo E2E` workflow sets `VERIFY_LOCAL_BUILD=standalone`, so its local devices run the same kind of app.
 
 ## Troubleshooting
 

.claude/skills/verify-clerk-expo/references/remote.md

@@ -46,7 +46,7 @@ A run on such a build is not tied to a commit. Its `run.json` names the patch un
 
 While the session builds, `up` and `run` print each new line of the build output, after `build   |`.
 
-After you push an edit to one of those paths, `run` asks the same session to build the new commit, with no patch. The session builds it only when GitHub shows it as a later commit of the branch the session was started on. After a rebase or an amend, the pushed commit is not a later commit of that branch, and the build fails with `BUILD_FAILED`. Run `down`, then `up`. While the native fingerprint is unchanged, a JS edit puts a new bundle in the native build the session already has. A session starts from a native build that a `Verify end-to-end tests` run of the branch kept for the same fingerprint, when there is one, and builds natively otherwise; [Freshness](freshness.md) has the rule. Specs run from your working tree, so an edit to a spec needs no commit. A session that already holds the build for your commit is reused as it is.
+After you push an edit to one of those paths, `run` asks the same session to build the new commit, with no patch. The session builds it only when GitHub shows it as a later commit of the branch the session was started on. After a rebase or an amend, the pushed commit is not a later commit of that branch, and the build fails with `BUILD_FAILED`. Run `down`, then `up`. While the native fingerprint is unchanged, a JS edit puts a new bundle in the native build the session already has. A session starts from a native build that an `Expo E2E` run of the branch kept for the same fingerprint, when there is one, and builds natively otherwise; [Freshness](freshness.md) has the rule. Specs run from your working tree, so an edit to a spec needs no commit. A session that already holds the build for your commit is reused as it is.
 
 A session keeps the `@clerk/internal-mobile-e2e` it installed when it started. When the version installed in your checkout differs from the session's, `up` and `run` fail with `NOT_READY`. The fix is to commit and push the new pin, then `down`, then `up`. The session reads `src/fixture.ts` and `src/platform/session-device.ts` again from each commit it builds. The build key leaves out everything under `.claude/`, so a commit that changes only those two files does not make a held session rebuild. After such a commit, run `down`, then `up`.
 

.github/workflows/verify-e2e.yml

@@ -1,19 +1,9 @@
-name: Verify end-to-end tests
+name: Expo E2E
 
 on:
   pull_request:
-    types: [opened, synchronize, reopened, ready_for_review]
     branches:
       - main
-    paths:
-      - 'integration/expo-native/**'
-      - '!integration/expo-native/**/*.md'
-      - '.github/workflows/verify-e2e.yml'
-      - 'integration/templates/expo-native/**'
-      - 'packages/expo/**'
-      - 'packages/expo-biometrics/**'
-      - 'packages/expo-google-signin/**'
-      - 'packages/expo-native-components/**'
   workflow_dispatch:
 
 permissions:
@@ -24,42 +14,9 @@ concurrency:
   cancel-in-progress: true
 
 jobs:
-  plan:
-    if: ${{ github.head_ref != 'changeset-release/main' }}
-    runs-on: ubuntu-latest
-    timeout-minutes: 5
-    outputs:
-      run: ${{ steps.plan.outputs.run }}
-    steps:
-      - name: Decide whether the end-to-end tests can run
-        id: plan
-        env:
-          HEAD_REPOSITORY: ${{ github.event.pull_request.head.repo.full_name || github.repository }}
-          HAS_PLATFORM_API_KEY: ${{ secrets.MOBILE_VERIFICATION_PLATFORM_API_KEY != '' }}
-          DRAFT: ${{ github.event.pull_request.draft == true }}
-        run: |
-          set -euo pipefail
-
-          skip_reason=""
-          if [ "$DRAFT" = "true" ]; then
-            skip_reason="The end-to-end tests did not run. This pull request is a draft. They run when it is marked ready for review."
-          elif [ "$HEAD_REPOSITORY" != "$GITHUB_REPOSITORY" ]; then
-            skip_reason="The end-to-end tests did not run. They need a repository secret, and this pull request comes from a fork ($HEAD_REPOSITORY)."
-          elif [ "$HAS_PLATFORM_API_KEY" != "true" ]; then
-            skip_reason="The end-to-end tests did not run. The MOBILE_VERIFICATION_PLATFORM_API_KEY secret is not set for this repository."
-          fi
-
-          if [ -n "$skip_reason" ]; then
-            echo "::notice title=End-to-end tests skipped::$skip_reason"
-            echo "run=false" >> "$GITHUB_OUTPUT"
-          else
-            echo "run=true" >> "$GITHUB_OUTPUT"
-          fi
-
   e2e-tests:
-    name: Verify E2E (${{ matrix.platform }})
-    needs: plan
-    if: needs.plan.outputs.run == 'true'
+    name: Device tests (${{ matrix.platform }})
+    if: ${{ github.event_name == 'workflow_dispatch' || (github.head_ref == 'changeset-release/main' && github.event.pull_request.head.repo.full_name == github.repository) }}
     runs-on: ${{ matrix.runner }}
     timeout-minutes: 75
     permissions:
@@ -71,9 +28,9 @@ jobs:
       matrix:
         include:
           - platform: ios
-            runner: ${{ vars.VERIFY_CI_RUNNER_IOS || 'macos-26' }}
+            runner: ${{ vars.VERIFY_CI_RUNNER_IOS || 'blacksmith-6vcpu-macos-26' }}
           - platform: android
-            runner: ${{ vars.VERIFY_CI_RUNNER_ANDROID || 'ubuntu-24.04' }}
+            runner: ${{ vars.VERIFY_CI_RUNNER_ANDROID || 'blacksmith-8vcpu-ubuntu-2204' }}
     env:
       PACKAGE: integration/expo-native
       CLI: integration/expo-native/bin/control-clerk-expo
@@ -92,7 +49,7 @@ jobs:
       - uses: actions/setup-node@v4.4.0
         with:
           node-version: 24.15.0
-          cache: ${{ matrix.platform == 'android' && 'pnpm' || '' }}
+          cache: pnpm
 
       - name: Set up JDK 21
         if: matrix.platform == 'android'
@@ -263,3 +220,17 @@ jobs:
             ${{ env.PACKAGE }}/.verify/runs/*/e2e*/artifacts
           retention-days: 3
           if-no-files-found: warn
+
+  verify-e2e:
+    name: Expo E2E
+    needs: e2e-tests
+    if: ${{ !cancelled() }}
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    steps:
+      - name: Pass when the end-to-end tests passed or did not need to run
+        env:
+          RESULT: ${{ needs.e2e-tests.result }}
+        run: |
+          echo "The end-to-end tests ended with $RESULT."
+          [ "$RESULT" = "success" ] || [ "$RESULT" = "skipped" ]