Skip to content

fix(ci): build website from latest benchmark results - #33

Open
rlorenzo wants to merge 1 commit into
lissy93:mainfrom
rlorenzo:fix/site-charts-use-results-branch
Open

fix(ci): build website from latest benchmark results#33
rlorenzo wants to merge 1 commit into
lissy93:mainfrom
rlorenzo:fix/site-charts-use-results-branch

Conversation

@rlorenzo

@rlorenzo rlorenzo commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Thanks for merging #26 (Gea)! While looking at it I noticed the comparison page isn't showing results for Gea or Astro: selecting either on its own leaves every chart empty. This PR is an attempt to fix that.

Cause

The site builds its charts from results/summary.json on main, which has been frozen since 2025-09-09, so anything added after that date renders empty.

benchmark.yml sets COMMIT_TO_MAIN=false on scheduled runs, and transform-results.yml gates its "Commit to Main Branch" step on that flag. The results branch is therefore refreshed nightly while main's copy only updates on a manual workflow_dispatch with commit_to_main: true. build.yml builds from main.

The chain is results/summary.jsonbuild_charts.py:981website/static/chart-configs.jsongenerator.py:204homepage.html:247 (const chartConfigs). The deployed chart labels match main's stale snapshot exactly: Alpine Angular Jquery Lit Preact Qwik React Solid Svelte Vanilla Vanjs Vue Lume-Js, with no Astro and no Gea.

Regression

Two commits on 2025-08-24:

139f504  Commits results to the results branch
0291354  Allows benchmarking to run as a cron, and skip commiting of results

Cron runs skipping main is clearly deliberate, and this PR preserves that. The unintended half is that the website build was never repointed at the results branch. The automated chore(results): update benchmark summaries from run #N commits stop at run #22 on 2025-08-23, the day before.

What follows looks like manual patching: be29a52 (2025-08-29) and df65ea9 (2025-09-09) hand-commit the exact files the pipeline had stopped writing. Since then the only change to results/summary.json was 77d721f (2026-05-09), which edited Lume.js's numbers into the old snapshot without touching generated_at. That is why Lume.js renders and Astro does not.

Fix

Sync summary/summary.json from the results branch into the build working tree before generating the site. Nothing is committed, so the cron policy is unchanged. If the branch is unreachable (forks), it warns and falls back to the committed file.

Verified:

  • Synced data gives generated: 2026-08-04 and 14 frameworks including Astro, against 13 and no Astro committed.
  • Re-ran build_charts.py: bundle_size_comparison and performance_radar now include Astro.
  • Fallback path tested against a missing branch: warning, exit 0, builds from the committed file.

Notes

Gea has never run in CI, since #26 merged 45 minutes after that day's cron had already checked out main. Because geajs is in frameworks.json and common.py:27 reads it unfiltered, the next scheduled run picks it up and this change carries it through to the site. Its CI build is therefore still unproven.

The Workflow Audit failure is pre-existing: 10 ref-version-mismatch warnings on uses: pins this diff does not touch, two of them in benchmark.yml. That job only runs when a workflow file changes, so it will fail any such PR until the pin comments are refreshed.


Disclosure: this PR was investigated and drafted with AI assistance (Claude Code).

The website job generates charts from results/summary.json on main, but
scheduled Benchmark runs only publish to the `results` branch: benchmark.yml
forces COMMIT_TO_MAIN=false on schedule, so transform-results.yml skips its
"Commit to Main Branch" step. main's copy therefore only updates on a manual
workflow_dispatch with commit_to_main=true.

The last such run was 2025-09-09, so the deployed site has been built from an
11-month-old snapshot. Every framework added since renders with empty charts -
currently Astro, and Gea once it has been benchmarked.

Sync summary/summary.json from the results branch into the build working tree
before generating the site. Nothing is committed, so the existing commit_to_main
policy is untouched; the deploy just stops reading a stale snapshot. Falls back
to the committed file with a warning annotation when the results branch is not
reachable, so forks are unaffected.
@rlorenzo

rlorenzo commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

The failing 🛠️ Workflow Audit isn't from this PR. All 10 ref-version-mismatch annotations sit on uses: lines the diff doesn't touch, including 2 in benchmark.yml, which it doesn't modify at all.

The pins are correct; the version comments are stale. actions/checkout@9c091bb2 is v7.0.0 and actions/setup-node@48b55a01 is v6.4.0, but the floating v7 and v6 tags have since moved, and zizmor compares the comment against what the ref resolves to now. On main this affects 39 pins across 11 workflows. It only surfaced here because the audit job runs only when a workflow file changes.

Comment-only fix, no pin or behaviour change:

-# v7
+# v7.0.0
-# v6
+# v6.4.0

Want that as a separate PR, pushed onto this branch, or left alone?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant