ci: publish the evidence a borrowed-device session hands off

[mikepitre]

What & why

A cloud agent cannot put a run's video and screenshots on its pull request. Its sandbox refuses the upload, and the Actions token cannot attach files. So attach in #1069 hands the files to the session's runner, which keeps them as the artifact verify-evidence. This PR adds the workflow that reads that artifact and puts the evidence in the pull request description, with a bot token.

It is the fifth PR of the stack and sits on #1069, which sits on #1052, #1046, and #1041. GitHub starts the workflow only from the copy on main. So it does nothing until this PR has merged, which is after the rest of the stack.

One commit.

File	Lines added	What it does
.github/workflows/verify-attach.yml	84	Runs on workflow_run when a verify-remote session ends, from the default branch.
.github/scripts/verify-attach.mjs	813	Downloads the artifact, validates it, and edits one block of the description with gh pr edit --attach.
.github/scripts/verify-attach.test.mjs	1,459	143 tests of the script and of the workflow file.
.github/workflows/android-test.yml	6	Runs those tests in the e2e-runner-tests job.

The script is the same file, byte for byte, in clerk/clerk-ios and clerk/javascript. Review it once.

• The artifact was written by branch code, so the job treats it as data. It never checks out, installs, or runs anything from the branch or from the artifact. The script reads the zip itself and refuses anything outside a fixed manifest, file names, sizes, counts, and leading bytes.
• It publishes only to the open pull request of the session's branch, in this repository, and only when the commit the session started on and the commit the run was made at are both commits of that pull request. GitHub lists only the first 250 commits of a pull request, and when a longer one has either commit past them the workflow says so and publishes nothing.
• The line it writes says that the session reported the result, links the session's run, and names the commit and the account that started the session, as GitHub records it. When the pull request has newer commits, the line says so. A later run and attach replaces the block.
• The block sits between the comments <!-- verify-evidence:android --> and <!-- /verify-evidence:android -->. Everything outside it is kept as it was, with the line endings the description has. The workflow refuses a description that has neither comment and ends inside a code fence that never closes.
• The runs for one branch take turns, and queue: max keeps every run that waits.

The environment verify-evidence exists in this repository and allows only main. It holds the secret VERIFY_EVIDENCE_TOKEN, a fine-grained token of a machine account that is limited to this repository and has "Pull requests: read and write". There is no repository secret of that name, and there must not be one.

A repository secret can be read by a workflow that anyone with write access adds on any branch. A secret of an environment that allows only main cannot. Without the secret the job prints a notice and publishes nothing.

What to focus on

• The token. The workflow has actions: read and contents: read, and only the step that runs gh holds the token. Check that nothing from the artifact or the branch can reach that step.
• The rule for which pull request gets the evidence, in pullRequestMismatch and commitNotListed.
• The workflow starts after every verify-remote session, with or without evidence, and each start shows as a deployment to verify-evidence.

node --test .github/scripts/verify-attach.test.mjs passes 143 of 143. Every rule has a test with an input that is refused. actionlint reports one finding in the new workflow, the queue key, which the installed version does not know.

Not proven, and not provable before the stack is on main:

• That a real session's end starts the workflow, and that the job gets the environment's secret.
• That a fine-grained token with only "Pull requests: read and write" can attach files with gh pr edit --attach.
• A real verify-evidence artifact. The script was run against zips made on a Mac.
• That queue: max keeps every waiting run in this workflow.
• A real hand-off from a cloud sandbox through the tunnel, which is feat(verify): borrow an emulator on a CI runner when the machine cannot run one #1069's part.

Screenshots / video

None. The change has no screen.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features
  • Verify-run evidence is validated and added to the relevant pull request, making results and supporting files easier to review.
  • When matching evidence is already present, the pull request can be updated with a note about newer commits.
• Bug Fixes
  • Evidence is not published when the archive is malformed or inconsistent, or the pull request no longer matches the verify run.
  • Evidence handling is skipped when the required token is unavailable.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: clerk/clerk-android/.coderabbit.yaml
• Review profile: CHILL
• Plan: Team
• Run ID: f855a366-3f50-4191-9ee6-3a8bc29e21ab

📥 Commits

Reviewing files that changed from the base of the PR and between d3f42d0 and 83c76d6.

📒 Files selected for processing (2)

• .github/scripts/verify-attach.test.mjs
• .github/workflows/verify-attach.yml

💤 Files with no reviewable changes (2)

• .github/workflows/verify-attach.yml
• .github/scripts/verify-attach.test.mjs

Included review availability: This review used your included allowance. 3 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 10 reviews per hour.

📝 Walkthrough

The pull request adds a script and workflow to fetch, validate, and publish evidence from eligible verify runs. It also adds tests for bundle validation, artifact fetching, publication behavior, and workflow configuration.

Changes: Verify evidence publication

Layer / File(s)	Summary
Evidence bundle validation .github/scripts/verify-attach.mjs, .github/scripts/verify-attach.test.mjs	The script validates ZIP structure, manifest fields, file names, sizes, checksums, and media signatures. Tests cover accepted bundles and rejection cases.
Artifact fetching and workflow setup .github/scripts/verify-attach.mjs, .github/scripts/verify-attach.test.mjs, .github/workflows/verify-attach.yml, .github/workflows/android-test.yml	The fetch step selects an eligible artifact and writes validated files to a new or empty work directory. The workflow gates runs, supplies run metadata, and conditionally runs the script tests when watched paths change.
Pull request evidence publication .github/scripts/verify-attach.mjs, .github/scripts/verify-attach.test.mjs, .github/workflows/verify-attach.yml	The publish step checks pull request and commit state, then adds or replaces evidence in the pull request description. It rechecks pull request state before editing. Tests cover evidence formatting, refusal paths, outputs, and workflow configuration.

Priority: ⬇️ Low

Estimated code review effort: 4 (Complex) | ~60 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant VerifyRun as verify-remote run
  participant Workflow as verify-attach workflow
  participant Script as verify-attach script
  participant ActionsAPI as GitHub Actions API
  participant GitHubCLI as GitHub CLI
  participant PullRequest as pull request
  VerifyRun-->>Workflow: Completed run metadata
  Workflow->>Script: Fetch evidence for the run
  Script->>ActionsAPI: List and download eligible artifact
  ActionsAPI-->>Script: Artifact ZIP
  Script->>Script: Validate bundle and write evidence files
  Workflow->>Script: Publish with run details and token
  Script->>GitHubCLI: Read pull request and commits
  GitHubCLI-->>Script: Pull request state and commit list
  Script->>GitHubCLI: Update pull request description
  GitHubCLI->>PullRequest: Save evidence block

Loading

Merge Risk: ⚪ Minimal · up to 83c76

No actionable issue was found that would block merging. The workflow still needs its normal first-run validation after it reaches the default branch.

Pre-merge checks: 4 · 1

Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 29 functions across 2 files. (1 skipped: 1…	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Title check		The title clearly and concisely describes the main change: publishing evidence from a borrowed-device session.
Description check		The description includes all template sections and provides detailed context, implementation scope, review risks, testing results, known limitations, and screenshots information.

Full details: Docstring Coverage

Explanation

Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 29 functions across 2 files. (1 skipped: 1 unsupported.)

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[swolfand]

• verify-attach.yml:29 uses concurrency: queue: max. I couldn't confirm GitHub accepts a queue key, and no other workflow here or in clerk-ios/clerk/javascript uses it. If it's invalid, the workflow file is rejected and, since it only runs from main, nobody notices until after merge. Can you confirm with actionlint or a scratch repo?
• verify-attach.mjs:701-732: a human edit to the PR description between the re-read and gh pr edit could be overwritten. Short window, low risk.
• actions/checkout and setup-node are pinned to v4 (Node 20); other workflows use newer versions.
• Nit: the comment at verify-attach.yml:77-78 describes our own step (AGENTS.md).

[swolfand]

Pushed fixes for the review findings above:

• d3cb780 merge feat(verify): borrow an emulator on a CI runner when the machine cannot run one #1069's fixes (runner-label allowlist, plus everything from test(e2e): configure the e2e host from verify launch extras #1041–ci: run the end-to-end tests in the release workflow, and remove the Trailblaze suite #1052)
• ca6b53f ci(verify): bump checkout and setup-node in verify-attach to v7, as the other workflows use. This drops the SHA pins to match the repo's @v7 tag style. Re-pin if you want this token-holding job pinned.

Left concurrency.queue: max as is. GitHub's workflow syntax docs list queue (single default, max up to 100 pending) for both workflow-level and jobs.<job_id>.concurrency. They say it can't be combined with cancel-in-progress: true, and this job uses false. actionlint 1.7.12 (latest) still reports unexpected key "queue", but that's a known gap: rhysd/actionlint#657 is open. The repo doesn't run actionlint in CI. The comment at ~:77-78 was already removed in your 453c64b.