fix(ci): build website from latest benchmark results - #33
Open
rlorenzo wants to merge 1 commit into
Open
Conversation
The website job generates charts from results/summary.json on main, but scheduled Benchmark runs only publish to the `results` branch: benchmark.yml forces COMMIT_TO_MAIN=false on schedule, so transform-results.yml skips its "Commit to Main Branch" step. main's copy therefore only updates on a manual workflow_dispatch with commit_to_main=true. The last such run was 2025-09-09, so the deployed site has been built from an 11-month-old snapshot. Every framework added since renders with empty charts - currently Astro, and Gea once it has been benchmarked. Sync summary/summary.json from the results branch into the build working tree before generating the site. Nothing is committed, so the existing commit_to_main policy is untouched; the deploy just stops reading a stale snapshot. Falls back to the committed file with a warning annotation when the results branch is not reachable, so forks are unaffected.
Contributor
Author
|
The failing 🛠️ Workflow Audit isn't from this PR. All 10 The pins are correct; the version comments are stale. Comment-only fix, no pin or behaviour change: -# v7
+# v7.0.0
-# v6
+# v6.4.0Want that as a separate PR, pushed onto this branch, or left alone? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thanks for merging #26 (Gea)! While looking at it I noticed the comparison page isn't showing results for Gea or Astro: selecting either on its own leaves every chart empty. This PR is an attempt to fix that.
Cause
The site builds its charts from
results/summary.jsonon main, which has been frozen since 2025-09-09, so anything added after that date renders empty.benchmark.ymlsetsCOMMIT_TO_MAIN=falseon scheduled runs, andtransform-results.ymlgates its "Commit to Main Branch" step on that flag. Theresultsbranch is therefore refreshed nightly while main's copy only updates on a manualworkflow_dispatchwithcommit_to_main: true.build.ymlbuilds from main.The chain is
results/summary.json→build_charts.py:981→website/static/chart-configs.json→generator.py:204→homepage.html:247(const chartConfigs). The deployed chart labels match main's stale snapshot exactly:Alpine Angular Jquery Lit Preact Qwik React Solid Svelte Vanilla Vanjs Vue Lume-Js, with no Astro and no Gea.Regression
Two commits on 2025-08-24:
Cron runs skipping main is clearly deliberate, and this PR preserves that. The unintended half is that the website build was never repointed at the
resultsbranch. The automatedchore(results): update benchmark summaries from run #Ncommits stop at run #22 on 2025-08-23, the day before.What follows looks like manual patching: be29a52 (2025-08-29) and df65ea9 (2025-09-09) hand-commit the exact files the pipeline had stopped writing. Since then the only change to
results/summary.jsonwas 77d721f (2026-05-09), which edited Lume.js's numbers into the old snapshot without touchinggenerated_at. That is why Lume.js renders and Astro does not.Fix
Sync
summary/summary.jsonfrom theresultsbranch into the build working tree before generating the site. Nothing is committed, so the cron policy is unchanged. If the branch is unreachable (forks), it warns and falls back to the committed file.Verified:
generated: 2026-08-04and 14 frameworks including Astro, against 13 and no Astro committed.build_charts.py:bundle_size_comparisonandperformance_radarnow include Astro.Notes
Gea has never run in CI, since #26 merged 45 minutes after that day's cron had already checked out main. Because
geajsis inframeworks.jsonandcommon.py:27reads it unfiltered, the next scheduled run picks it up and this change carries it through to the site. Its CI build is therefore still unproven.The Workflow Audit failure is pre-existing: 10
ref-version-mismatchwarnings onuses:pins this diff does not touch, two of them inbenchmark.yml. That job only runs when a workflow file changes, so it will fail any such PR until the pin comments are refreshed.Disclosure: this PR was investigated and drafted with AI assistance (Claude Code).