Skip to content

setup-chromatic: build without a token and publish from the default branch - #50

Open
CodyCBakerPhD wants to merge 6 commits into
masterfrom
claude/happy-cori-ke2ppt
Open

CodyCBakerPhD wants to merge 6 commits into
masterfrom
claude/happy-cori-ke2ppt

Conversation

@CodyCBakerPhD

@CodyCBakerPhD CodyCBakerPhD commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Lessons learned from the usage page on DANDI; this allows Chromatic to run from forks without too much trouble

🤖 AI text blow 🤖

setup-chromatic now writes the three-workflow form dandi/usage-page uses, for every repository rather than only those taking pull requests from forks:

  • chromatic-storybook.yml and chromatic-playwright.yml build with no token on push (and on a fork's pull_request) and hand the build on as an artifact.
  • New chromatic-publish.yml uploads it on workflow_run from the default branch, with --exit-zero-on-changes, and --auto-accept-changes only for a push to the default branch.
    Forks' pull requests get snapshots, and merges no longer leave builds waiting for review in Chromatic (and notifications about them).
  • The publish template holds the tokens safely: persist-credentials: false, npm ci --ignore-scripts from the default branch, the built commit fetched by hash and never checked out (only HEAD and a stub package.json; in a checkout the CLI runs the repository's own Storybook and package-manager configuration with the token in reach), triggering-run values only through env:, each token referenced statically behind its own condition (a missing one stays empty; a computed secrets[...] name would send every secret to the runner), and auto-accept keyed to github.event.repository.default_branch.
  • references/fork-pull-requests.md becomes references/publish-workflow.md (why the split, those properties, what the three files rely on); step 0 drops the fork question; steps 6, 7, 10 and 11, the README template, troubleshooting and the collection README follow.

dandi/usage-page's own chromatic-publish.yml checks the commit out and wants the same fix.

…efault branch

setup-chromatic now writes the three-workflow form dandi/usage-page uses, for every
repository rather than only those taking pull requests from forks: the Storybook and
Playwright workflows build with no token and hand the build on as an artifact, and a new
chromatic-publish.yml uploads it on workflow_run from the default branch. Forks' pull
requests get snapshots, and a push to the default branch is accepted as the new baseline,
so merges no longer leave builds waiting for review. references/fork-pull-requests.md
becomes references/publish-workflow.md: why the split, the publish workflow's safety
properties (now in the template), and what the three files rely on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DQrzw3QuYxA1bV48ZgQcor
CodyCBakerPhD pushed a commit to brain-bbqs/bbqs-web-app-template that referenced this pull request Nov 19, 2026
The template's own setup-chromatic skill is removed; apm.yml installs it from con/skills
(pinned in apm.lock.yaml to the commit of con/skills#50, which carries the three-workflow
form), and `apm install --frozen` restores it into .claude/skills/, where it is git-ignored.
AGENTS.md says how to restore it and where fixes go, setup-web-app keeps the APM files and
renames apm.yml, and the three Chromatic workflows now match the skill's templates
(Node 22, persist-credentials: false, auto-accept keyed to the default branch), plus the
template's own publish guard. .prettierignore keeps Prettier off the APM-written lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DQrzw3QuYxA1bV48ZgQcor
@CodyCBakerPhD CodyCBakerPhD self-assigned this Nov 19, 2026
@CodyCBakerPhD
CodyCBakerPhD marked this pull request as ready for review November 19, 2026 21:07
CodyCBakerPhD added a commit to brain-bbqs/bbqs-web-app-template that referenced this pull request Nov 19, 2026
…tic from con/skills (#17)

* Published Chromatic builds the way usage-page does

The two Chromatic workflows now only build (Storybook, and the Playwright archive Storybook) with
no token and hand the result on as an artifact; a new chromatic-publish.yml uploads it from main
on workflow_run, auto-accepting builds from a push to main, so merges no longer leave builds
waiting for review and pull requests from forks get snapshots. The setup-chromatic skill's step 7
now writes these three files, and setup-web-app names the publish job's template guard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DQrzw3QuYxA1bV48ZgQcor

* Named the template by its real repository in the publish guard

GitHub reports this repository as brain-bbqs/bbqs-web-app-template (web-app-template only
redirects), which is what the other workflow guards already match.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DQrzw3QuYxA1bV48ZgQcor

* Installed setup-chromatic from con/skills instead of keeping a copy

The template's own setup-chromatic skill is removed; apm.yml installs it from con/skills
(pinned in apm.lock.yaml to the commit of con/skills#50, which carries the three-workflow
form), and `apm install --frozen` restores it into .claude/skills/, where it is git-ignored.
AGENTS.md says how to restore it and where fixes go, setup-web-app keeps the APM files and
renames apm.yml, and the three Chromatic workflows now match the skill's templates
(Node 22, persist-credentials: false, auto-accept keyed to the default branch), plus the
template's own publish guard. .prettierignore keeps Prettier off the APM-written lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DQrzw3QuYxA1bV48ZgQcor

---------

Co-authored-by: Claude <noreply@anthropic.com>

@yarikoptic-gitmate yarikoptic-gitmate left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @CodyCBakerPhD. The three-workflow form reads well and the skill is consistent after the change (no stale references to the action, the fork reference or the old question numbering). CI is green.

One blocker before this is merged: the publish workflow's central claim, that nothing from the built commit executes while the tokens are in reach, does not hold. The Chromatic CLI loads Storybook configuration and spawns the package manager from its working directory, which is the fork's checkout, so a fork's pull request can read the project token. This was reproduced locally on two CLI versions. The inline comment has a two-line fix (fetch the commit, do not check its files out), which was checked locally but still needs one real run. dandi/usage-page's own chromatic-publish.yml has the same problem and wants the same change.

Also inline:

  • Token selection falls back to the Storybook token when the Playwright secret is missing, so a Playwright build can land in, and become the baseline of, the Storybook project. One-line suggestion.
  • Troubleshooting: the Dependabot secrets advice looks wrong per GitHub's docs, and the "no Chromatic check" row lacks its likeliest cause.
  • README template: the first build is no longer accepted by hand.
  • TurboSnap: dropping it is reasonable, but the snapshot cost deserves a mention.

Verified and fine as they are: a fork's commit is still fetched by actions/checkout with fetch-depth: 0; actions/checkout could not replace the hand-driven fetch in the publish job (v7 refuses a fork's commit on workflow_run); the auto-accept guard; CHROMATIC_SHA/BRANCH/SLUG match what the Chromatic action sets; uploading the archive Storybook without --playwright is equivalent.

This review went through two independent reviewer passes (GitHub Actions security; Chromatic and skill authoring) before posting.


Generated by Claude Code

Run inside a checkout of the built commit, the Chromatic CLI loads its Storybook config and
starts the package manager it detects there, which honours repository-local configuration:
chromatic 18.9.4 ran a script a fork's .yarnrc named, with the project token in its environment,
before authenticating. The publish template now points HEAD at the commit with
`git update-ref --no-deref` and writes only a stub package.json. The token is chosen by the
secret's name, so a missing secret no longer falls through to the other project's token.
Review follow-ups: the troubleshooting rows (Dependabot advice dropped, the publish run as the
first cause of a missing check, a repository that is itself a fork), the README template's
first-baseline wording, and TurboSnap's snapshot cost; publish-workflow.md loses two paragraphs
that repeated step 7 and the README template.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DQrzw3QuYxA1bV48ZgQcor

@yarikoptic-gitmate yarikoptic-gitmate left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed c65afe5: all six points from the first round are addressed, nothing is left stale in the skill, and CI is green. Nothing blocking remains on the code side.

Still outstanding before merge: one real run of chromatic-publish.yml for a fork's pull request, since nothing here could go past the Chromatic API. The blocker thread stays open for that.

What was checked this round, by two independent reviewer passes (GitHub Actions security; Chromatic and skill authoring):

  • No-checkout fix, run locally with chromatic 18.9.4 and 18.15.1 against a mock of the Chromatic API, under strace, with a hostile commit (.yarnrc yarn-path, .npmrc script-shell, Storybook main.js, chromatic.config.json, .gitattributes filter, ext:: submodule URL):
    • With the old git checkout --detach the planted script ran with the token in its environment.
    • With git update-ref --no-deref HEAD and the stub package.json nothing from the commit ran. The CLI spawned only yarn/npm version and registry queries and read-only git commands, resolved the right commit, branch and parent build, and got as far as announcing the build.
    • Upload, verify and snapshot were not reached by the mock.
  • Token by name: an empty CHROMATIC_PROJECT_TOKEN stops the CLI with Missing project token before any API call.
  • From the CLI source only, not run: the git commands that write a worktree are behind --patch-build or TurboSnap, neither used here; and the stub also removes a second path, where the -c directory of a commit's build-storybook script is interpolated into a sh -c "git log …" call.
  • Repo conventions: setup-chromatic declares no version:, so there is no bump to make; ci/validate_skills.py passes for it.

One optional wording nit, in templates/README-visual-snapshots.md step 3: the colon joins "their changes were reviewed on their pull requests" to the first build, which is the one build that was not reviewed. Reads more clearly as:

   The first upload comes from the merge that adds these files; that first build of each project is its baseline, so look it over in Chromatic once.
   After that, the default branch's builds are accepted on their own, since their changes were reviewed on their pull requests.

Generated by Claude Code

A computed secrets[...] name makes GitHub send every repository and organization secret to
the runner (CodeQL: excessive secrets exposure, flagged on brain-bbqs/bbqs-web-app-template#19).
Each token now sits behind its own condition instead, which keeps a missing secret empty so the
CLI stops rather than uploading into the other project. Also takes the review's wording for the
README template's first-baseline step.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DQrzw3QuYxA1bV48ZgQcor
AGENTS.md called `version:` in SKILL.md frontmatter "the field APM reads". apm-cli 0.31's
SKILL.md parser takes only name and description from the frontmatter, and an install records
the package version from apm.yml (0.1.0 for every con/skills skill). The field stays what the
skills that stamp their output read it for.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DQrzw3QuYxA1bV48ZgQcor
Comment thread AGENTS.md Resolved
This reverts commit 8a74a96. #51 rewrites the
AGENTS.md "Skill versions" section, which covers this sentence, so this pull request stays
limited to setup-chromatic.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DQrzw3QuYxA1bV48ZgQcor
CodyCBakerPhD added a commit to NeurodataWithoutBorders/nwb-benchmarks-dashboard that referenced this pull request Nov 20, 2026
…lish job's token exposure (#15)

* Installed setup-chromatic from con/skills and aligned the Chromatic workflows with it

The repository's own copy of setup-chromatic is removed; apm.yml installs it from con/skills
(pinned in apm.lock.yaml to the commit of con/skills#50), and `apm install --frozen` restores it
into .claude/skills/, where it is git-ignored. AGENTS.md says how to restore it and where fixes
go, and .prettierignore keeps Prettier off the APM-written lock. The three Chromatic workflows are
now the skill's templates: Node 22, persist-credentials: false on the publish job's checkout, and
auto-accept keyed to the repository's default branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DQrzw3QuYxA1bV48ZgQcor

* Fetched the built commit without checking it out in the Chromatic publish job

Run inside a checkout of the built commit, the Chromatic CLI loads its Storybook config and
starts the package manager it detects there, which honours repository-local configuration, so a
fork's pull request could read the project tokens (reproduced with chromatic 18.9.4 and a
.yarnrc). The publish job now points HEAD at the commit and writes only a stub package.json,
and picks the token by the secret's name. Both come from setup-chromatic at con/skills c65afe5,
which apm.yml now pins.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DQrzw3QuYxA1bV48ZgQcor

* Referenced the two Chromatic tokens statically in the publish job

A computed secrets[...] name makes GitHub send every repository and organization secret to the
runner (CodeQL: excessive secrets exposure). Each token now sits behind its own condition, which
keeps a missing secret empty so the CLI stops rather than uploading into the other project.
From setup-chromatic at con/skills 71e788c, which apm.yml now pins.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DQrzw3QuYxA1bV48ZgQcor

---------

Co-authored-by: Claude <noreply@anthropic.com>
@CodyCBakerPhD

Copy link
Copy Markdown
Contributor Author

Ran chromatic-publish.yml for real on a fork's pull request, in NeurodataWithoutBorders/nwb-benchmarks-dashboard, whose three Chromatic workflows are byte-identical to this PR's templates at f6e8121.

Fork PR: NeurodataWithoutBorders/nwb-benchmarks-dashboard#17, from CodyCBakerPhD:claude/show-load-card-in-stories (head e33cb14, which exists only in the fork).

  • Both build workflows ran on pull_request, without secrets, and handed on their artifacts.
  • Storybook publish (run, Chromatic build 33) found 2 visual changes and left them pending.
  • Playwright publish (run, Chromatic build 31) found no changes.
  • Both logs show CHROMATIC_SLUG: CodyCBakerPhD/nwb-benchmarks-dashboard, the fork's branch, and AUTO_ACCEPT: false.
  • The fork-only commit was fetched by hash and never checked out, and the CLI (v18.9.4) still read its history: "Commit 'e33cb14' on branch 'claude/show-load-card-in-stories'; found 1 parent build".
  • The two runs authenticated with different project tokens, so each upload went to its own project.
  • All four Chromatic statuses (publish and UI Tests, for each project) landed on the PR's head commit.

The default-branch push path ran earlier, when dashboard#15 merged: publish #15 and publish #16.

@yarikoptic-gitmate yarikoptic-gitmate left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @CodyCBakerPhD. That real run was the last open item from my side, and what you report covers each thing the local test could not reach:

  • the fork-only commit fetched by hash, never checked out, and the CLI still finding its branch and parent build;
  • pending (not auto-accepted) changes for a fork's pull request, with AUTO_ACCEPT: false;
  • each run using its own project's token;
  • the statuses landing on the pull request's head commit;
  • the default-branch push path.

Checked here at f6e8121: the head differs from 71e788c, which I reviewed last, by a commit and its revert, so the tree is identical and the earlier review stands. All threads are resolved, CI is green and there is no merge conflict.

Not checked: the linked runs and Chromatic builds themselves. nwb-benchmarks-dashboard is outside the repositories I can read, so I take the run results, and the templates there being identical to this PR's, as reported.

Nothing further from this review. I am leaving it as a comment, not an approval; merging is for a maintainer to decide.


Generated by Claude Code

@CodyCBakerPhD

Copy link
Copy Markdown
Contributor Author

@yarikoptic I tested this skill in practice and while it mostly worked well, one edge case came up regarding how to deal with fork PRs, and this is a better overall way of handling that case

If you'd like to review yourself, or defer to the bots opinion (which is now satisfied) and just merge; this is slightly larger than a 'simple fix' so I didn't just go ahead and merge myself

Copy link
Copy Markdown
Contributor

Checked ccf0b64, the merge of master (#52) into this branch. It brings in only #52's changes to bot-pr-sweep (SKILL.md and scripts/pr_gate.py); skills/setup-chromatic/ is identical to f6e8121, which I reviewed last. CI is green on this commit.

The earlier review stands, with nothing open from my side. Merging is for a maintainer to decide.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants