Skip to content

feat(verify): borrow an emulator on a CI runner when the machine cannot run one - #1069

Draft
mikepitre wants to merge 5 commits into
mike/verify-ci-device-specs-androidfrom
mike/verify-borrowed-device-android
Draft

mikepitre wants to merge 5 commits into
mike/verify-ci-device-specs-androidfrom
mike/verify-borrowed-device-android

Conversation

@mikepitre

@mikepitre mikepitre commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

What & why

  • The end-to-end tests in feat(verify): add the end-to-end test package and the verify-clerk-android skill #1046 run on an emulator on the machine that runs their CLI. An agent in a cloud sandbox often cannot run one. This PR adds a second backend to that CLI. On a machine that cannot run the emulator, up starts a session on a GitHub Actions runner, and the CLI drives the emulator there through a tunnel, with the same verbs, tests, and evidence.
  • No session has been started from the present head of this branch, so the remote backend is covered by unit tests only.
  • The tests and the CLI are the package e2e-tests/, and the verify-clerk-android skill is the instructions that point an agent at it.
  • It sits on ci: run the end-to-end tests in the release workflow, and remove the Trailblaze suite #1052, and ci: publish the evidence a borrowed-device session hands off #1079, which adds the upload workflow, sits on it. Nothing in test(e2e): configure the e2e host from verify launch extras #1041, feat(verify): add the end-to-end test package and the verify-clerk-android skill #1046, or ci: run the end-to-end tests in the release workflow, and remove the Trailblaze suite #1052 needs it.
  • --backend remote on doctor, up, and run forces the remote backend. With no flag the CLI picks the local backend where the machine can run the emulator and the remote one anywhere else, and its first line says which and why.
  • .github/workflows/verify-remote.yml runs one session. It builds the pushed commit, never the working tree, boots the emulator, records the video, and ends on down, after 15 idle minutes, or at 60 minutes.
  • --runner <label> names the runner for a new session. The default is ubuntu-24.04.
  • With the remote backend, doctor checks that git can fetch and push the branch, that GitHub has HEAD, and that this machine reaches GitHub, the tunnel host, and Clerk. It starts nothing without --live. doctor --live starts one short session and stops it.
  • down ends the runner job, and down --stale also ends a session that a crashed run of the same checkout left running.
  • When the machine's gh cannot attach files, attach hands the video and screenshots to the session's runner through the tunnel, and the runner keeps them as the artifact verify-evidence.
  • In the skill directory, references/remote.md is new and SKILL.md describes the remote backend. Outside it and e2e-tests/, the PR adds verify-remote.yml and one line in android-test.yml, and it changes one sentence in AGENTS.md.

To try it, on any machine with Node 24.8 or newer, the team's Clerk Platform API key, and a GitHub token that can push to this repository and start a workflow in it:

$ npm ci --prefix e2e-tests
$ git push
$ e2e-tests/bin/control-clerk-android doctor --backend remote
$ e2e-tests/bin/control-clerk-android run auth-start --backend remote
$ e2e-tests/bin/control-clerk-android down

On a machine that cannot run the emulator, leave out --backend remote. Allow about ten minutes for the first up, most of it the first Gradle build.

What to focus on

Five commits, in dependency order.

Commit Lines What it does
1. Evidence in the description 397 added, 100 removed Makes attach write a run's video and screenshots into the pull request description, in one block that a later attach replaces. In #1046 it posts a comment.
2. The remote session 2,111 added Adds the workflow that boots an emulator on a GitHub runner for one session, the code on the runner that drives the emulator, and the code on your machine that talks to it through a tunnel.
3. The choice of backend 293 added, 82 removed Teaches the CLI to choose between an emulator on this machine and one on a runner, and tells the tests' support code how to reach a borrowed one. Outside the unit tests, commits 1 and 3 and two small edits in commit 2 hold every edit to code that #1046 added.
4. Unit tests 2,078 added, 7 removed Tests of the borrowed-device code.
5. Docs 142 added, 9 removed Tells an agent how a session works, what it costs, and how to end it.

Commit 1 has its own unit tests. The unit tests for commits 2 and 3 arrive in commit 4.

Everything under e2e-tests/src/core/ and e2e-tests/specs/support/, and most of the unit tests, are the same files, byte for byte, in clerk/clerk-ios#661 and clerk/javascript#10131. src/core/MANIFEST lists every shared file with its hash, and a unit test fails when one drifts. Review them once.

Not proven:

  • A session started from this head. No session has run from these commits, and none has run the present 19 tests.
  • doctor --live, the idle stop, and the 60-minute cap, at this head.

What limits the risk:

  • A session starts by workflow_dispatch only, so it takes write access to start one and a fork cannot. The workflow has no push or pull_request trigger and uses no repository or environment secret. Its one permissions block is contents: read, and no job has a write permission. The step that reads the request uses the job's own read-only token to check the requested commit. A session starts only for the commit the run was dispatched on, or for a later commit of that branch.
  • Every route on the tunnel answers 403 without a bearer token that the CLI made. The token stays under e2e-tests/.verify/remote/ on the machine that started the session. The workflow gets only its SHA-256.
  • The tunnel is a Cloudflare quick tunnel, so Cloudflare can read the traffic, which includes sign-in tickets for test users. Clerk's keys stay on the machine that drives the session, and down deletes the throwaway Clerk application.
  • A session costs nothing on the default ubuntu-24.04 label. A Blacksmith label is billed by the minute.
  • A run on the machine's own emulator takes the same path as before this PR. Apart from attach, the new code runs only when the CLI picks the remote backend. The release gate in ci: run the end-to-end tests in the release workflow, and remove the Trailblaze suite #1052 does not use it.

npm test --prefix e2e-tests passes 665 of 665 at this head with no network, key, or emulator, and npm run typecheck --prefix e2e-tests reports nothing.

Screenshots / video

None attached.

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@mikepitre
mikepitre force-pushed the mike/verify-borrowed-device-android branch from a93327c to f36fa01 Compare October 8, 2026 04:36
@mikepitre
mikepitre force-pushed the mike/verify-borrowed-device-android branch from f36fa01 to 3adb90e Compare October 8, 2026 12:02
@mikepitre
mikepitre force-pushed the mike/verify-borrowed-device-android branch from 3adb90e to b426eed Compare October 8, 2026 14:19
@mikepitre
mikepitre force-pushed the mike/verify-borrowed-device-android branch from b426eed to 55b313f Compare October 8, 2026 20:40
@mikepitre
mikepitre force-pushed the mike/verify-borrowed-device-android branch from 55b313f to e54b8b5 Compare October 8, 2026 20:45
@mikepitre
mikepitre force-pushed the mike/verify-borrowed-device-android branch from e54b8b5 to a0da184 Compare October 8, 2026 20:49
@mikepitre
mikepitre added this pull request to stack #1053 October 8, 2026 22:12
@mikepitre
mikepitre force-pushed the mike/verify-borrowed-device-android branch from a0da184 to 5558f6b Compare October 9, 2026 01:04
@mikepitre
mikepitre force-pushed the mike/verify-borrowed-device-android branch 2 times, most recently from f699ca9 to 026ffb2 Compare October 9, 2026 03:57
@mikepitre
mikepitre force-pushed the mike/verify-borrowed-device-android branch from 026ffb2 to ce42c02 Compare October 9, 2026 05:23
@mikepitre
mikepitre force-pushed the mike/verify-borrowed-device-android branch from ce42c02 to 184115f Compare October 9, 2026 14:52
mikepitre and others added 2 commits October 9, 2026 14:40
`attach` posted the video and screenshots of a run as a pull request
comment. It now writes them into the description with
`gh pr edit --attach`, in one block between two HTML comments that name
the platform. A later `attach` replaces the block, so the description
holds the latest run. Everything outside the block is passed back as it
was read.

`attach` reads the description again just before it writes. If the
description changed, it rebuilds on the new text once, and fails if it
changed again. It refuses a description whose markers are doubled,
halved, or out of order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mikepitre and others added 3 commits October 9, 2026 14:40
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mikepitre
mikepitre force-pushed the mike/verify-borrowed-device-android branch from 184115f to 663183b Compare October 9, 2026 19:17

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant