Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/benchmark.yml
Original file line number Diff line number Diff line change
Expand Up @@ -52,15 +52,15 @@ jobs:
tool: 'customSmallerIsBetter'
output-file-path: benchmark-gh.json
gh-pages-branch: 'gh-pages'
benchmark-data-dir-path: 'benchmarks'
benchmark-data-dir-path: 'metrics'
github-token: ${{ secrets.GITHUB_TOKEN }}
fail-on-alert: false
alert-threshold: '200%'
comment-on-alert: true
auto-push: ${{ github.event_name == 'push' }}

publish-pages:
name: Publish updated benchmarks
name: Publish updated metrics
needs: benchmark
if: github.event_name == 'push'
permissions:
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,7 @@ jobs:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7

- name: Test benchmark dashboard data handling
run: node --test tests/benchmark_metrics.test.mjs
run: node --test tests/metrics_dashboard.test.mjs

- name: Install uv
uses: astral-sh/setup-uv@bec219d24cd3e171d82865faccec33120bb574f4 # v10.1.0
Expand Down Expand Up @@ -180,7 +180,7 @@ jobs:
tool: 'customSmallerIsBetter'
output-file-path: benchmark-gh.json
gh-pages-branch: 'gh-pages'
benchmark-data-dir-path: 'benchmarks'
benchmark-data-dir-path: 'metrics'
github-token: ${{ secrets.GITHUB_TOKEN }}
fail-on-alert: true
alert-threshold: '250%'
Expand Down
10 changes: 5 additions & 5 deletions .github/workflows/mutation.yml
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,8 @@ jobs:
run: |
if [[ "$EVENT" == "schedule" ]]; then
git fetch origin gh-pages
if git cat-file -e origin/gh-pages:benchmarks/mutation-history.json; then
git show origin/gh-pages:benchmarks/mutation-history.json > mutation-history.json
if git cat-file -e origin/gh-pages:metrics/mutation-history.json; then
git show origin/gh-pages:metrics/mutation-history.json > mutation-history.json
fi
changed="$(python3 scripts/mutation_report.py changed --history mutation-history.json)"
else
Expand Down Expand Up @@ -246,9 +246,9 @@ jobs:
echo "Publishing mutation results (attempt $attempt/5)."
python3 scripts/mutation_report.py record \
--results results --matrix "$MATRIX" --commit "$COMMIT" --date "$recorded_at" \
--history pages-data/benchmarks/mutation-history.json \
--latest pages-data/benchmarks/mutation-latest.json
git -C pages-data add benchmarks/mutation-history.json benchmarks/mutation-latest.json
--history pages-data/metrics/mutation-history.json \
--latest pages-data/metrics/mutation-latest.json
git -C pages-data add metrics/mutation-history.json metrics/mutation-latest.json
if ! git -C pages-data diff --cached --quiet; then
git -C pages-data commit -m "Record engine mutation score for ${COMMIT:0:7}"
fi
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/pages.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ jobs:
fetch-depth: 0
persist-credentials: false

- name: Read current published versions and benchmarks
- name: Read current published versions and metrics
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: gh-pages
Expand All @@ -44,15 +44,15 @@ jobs:
- name: Install documentation dependencies
run: uv sync --only-group docs --locked

- name: Assemble documentation, redirects, and benchmarks
- name: Assemble documentation, redirects, and metrics
run: |
latest_version="$(uv run --only-group docs python scripts/prepare_pages.py pages-source --print-latest)"
git show "refs/tags/v${latest_version}:zensical.toml" > latest-zensical.toml
uv run --only-group docs python scripts/prepare_pages.py pages-source \
--output published --config latest-zensical.toml

- name: Test benchmark dashboard data handling
run: node --test tests/benchmark_metrics.test.mjs
- name: Test metrics dashboard data handling
run: node --test tests/metrics_dashboard.test.mjs

- name: Upload Pages artifact
uses: actions/upload-pages-artifact@fc324d3547104276b827a68afc52ff2a11cc49c9 # v5.0.0
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -204,7 +204,7 @@ jobs:
git push origin gh-pages

publish-pages:
name: Publish documentation and benchmarks
name: Publish documentation and metrics
needs: build-docs
if: ${{ !cancelled() && needs.build-docs.result == 'success' }}
permissions:
Expand Down
8 changes: 4 additions & 4 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -83,12 +83,12 @@ repos:

# --- Pre-push hooks --- #

- id: benchmark-dashboard
name: benchmark dashboard data handling
entry: node --test tests/benchmark_metrics.test.mjs
- id: metrics-dashboard
name: metrics dashboard data handling
entry: node --test tests/metrics_dashboard.test.mjs
language: system
pass_filenames: false
files: ^(benchmarks/|tests/benchmark_metrics\.test\.mjs$)
files: ^(metrics/|tests/metrics_dashboard\.test\.mjs$)
stages: [pre-push]

- id: zensical-build
Expand Down
6 changes: 3 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -195,11 +195,11 @@ just test

### Documentation Publishing

GitHub Pages must use **GitHub Actions** as its source (Settings → Pages → Build and deployment). The `gh-pages` branch stores released documentation and benchmark data; it is not itself the published site. Releases and benchmark updates invoke `.github/workflows/pages.yml`, which serializes site assembly and deployment together and reads the current branch contents after acquiring its publishing slot.
GitHub Pages must use **GitHub Actions** as its source (Settings → Pages → Build and deployment). The `gh-pages` branch stores released documentation and benchmark and mutation data; it is not itself the published site. Releases and benchmark updates invoke `.github/workflows/pages.yml`, which serializes site assembly and deployment together and reads the current branch contents after acquiring its publishing slot.

The publisher creates a clean artifact with `scripts/prepare_pages.py`. The release carrying `latest` in `versions.json` is served at the bare documentation URLs, and every release, that one included, keeps its versioned copy. Each versioned page the latest release still has names the bare URL as its canonical address, so search engines rank one URL across releases; pages the latest release dropped are marked `noindex`. Version aliases redirect to wherever their release is served, preserving query parameters and anchors, and the sitemap lists the bare URLs. Old root copies are excluded. Markdown and `llms.txt` come from the latest release's HTML and tagged navigation, with links to the bare Markdown files. Benchmark history is included in every deployment. The root also gets `robots.txt`, which welcomes search engines and AI search agents, turns away AI training crawlers, and names the sitemap.
The publisher creates a clean artifact with `scripts/prepare_pages.py`. The release carrying `latest` in `versions.json` is served at the bare documentation URLs, and every release, that one included, keeps its versioned copy. Each versioned page the latest release still has names the bare URL as its canonical address, so search engines rank one URL across releases; pages the latest release dropped are marked `noindex`. Version aliases redirect to wherever their release is served, preserving query parameters and anchors, and the sitemap lists the bare URLs. Old root copies are excluded. Markdown and `llms.txt` come from the latest release's HTML and tagged navigation, with links to the bare Markdown files. The metrics dashboard is included in every deployment, and the old `/benchmarks/` address redirects to it. The root also gets `robots.txt`, which welcomes search engines and AI search agents, turns away AI training crawlers, and names the sitemap.

To republish the current documentation and benchmark data without rebuilding a release or publishing to PyPI, run `gh workflow run pages.yml --ref main`. To rebuild a released documentation version, run `gh workflow run release.yml --ref main -f tag=vX.Y.Z`; this moves `latest` to that version and invokes the same publisher. Keep Pages in Actions mode so branch updates cannot replace the assembled artifact.
To republish the current documentation and metrics without rebuilding a release or publishing to PyPI, run `gh workflow run pages.yml --ref main`. To rebuild a released documentation version, run `gh workflow run release.yml --ref main -f tag=vX.Y.Z`; this moves `latest` to that version and invokes the same publisher. Keep Pages in Actions mode so branch updates cannot replace the assembled artifact.

### Isolated Manual Testing (Sandboxes)

Expand Down
4 changes: 2 additions & 2 deletions MUTATION_TRACKING_PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ Widen mutation testing to the rest of the engine, then run it on a schedule, rec
- **Scheduled, gated by change.** No run on every commit. A scheduled run first checks whether the mutated modules, `tests/`, `pyproject.toml`, `uv.lock`, `scripts/mutation_report.py`, or the mutation workflow changed since the last recorded commit, and stops if not. Quiet periods cost nothing, and a busy period gets one point per scheduled run. The manual trigger stays for development.
- **Only complete runs publish.** A run publishes a score only if every module in the set succeeded. Manual runs on a subset never publish.
- **Store raw counts per module.** Each history entry records the commit, the date, and each module's counts (killed, timeout, survived, and the rest `mutation_report.py` tracks), not just percentages. The overall score is computed from them, so it stays correct when the set grows, and the graph can mark when the scope changed.
- **Our own JSON, not github-action-benchmark.** That action writes a JS file built for speed regressions. Mutation results go in their own files under `benchmarks/` on `gh-pages`: a history file for the graph and a small latest-score file in shields.io's endpoint format, which the badge and the website both read.
- **Our own JSON, not github-action-benchmark.** That action writes a JS file built for speed regressions. Mutation results go in their own files under `metrics/` on `gh-pages`: a history file for the graph and a small latest-score file in shields.io's endpoint format, which the badge and the website both read.
- **The website reads the score when it builds.** jacksonferguson.me (Astro, `~/Developer/JacksonFergusonDev.github.io`) already fetches remote files at build time through `config/remote-assets.json`, and its `deploy.yml` already rebuilds on `repository_dispatch` of type `remote-assets-updated`, plus weekly. Protostar sends that dispatch after the Pages deploy finishes, not just after the `gh-pages` push, so the site never fetches a stale file.
- **Say what the score covers.** It covers the mutated modules, not the whole codebase. The dashboard says which modules, and the badge label doesn't imply full coverage.

Expand Down Expand Up @@ -62,5 +62,5 @@ Added so far:
| `reconciliation` (6 shards) | 2h30m wall, 8h40m runner | 36m wall, 1h55m runner |

- Phase 3 (#441): Complete main-branch runs publish raw per-module history and the latest-score endpoint nightly at 02:23 UTC, gated by changed inputs. Pages carries both JSON files.
- Phase 4 (#442): The benchmarks dashboard graphs the overall score and each module from `mutation-history.json`, marking where the module set changed, and the README carries a shields.io endpoint badge reading `mutation-latest.json`.
- Phase 4 (#442): The metrics dashboard graphs the overall score and each module from `mutation-history.json`, marking where the module set changed, and the README carries a shields.io endpoint badge reading `mutation-latest.json`.
- Phase 5 (#443, JacksonFergusonDev.github.io#7): jacksonferguson.me fetches `mutation-latest.json` at build time through a `json` remote asset type that checks the file's shape, and shows the score on the Protostar card, linked to the dashboard. After a published run's Pages deploy, the mutation workflow's `refresh-site` job sends the site a `remote-assets-updated` dispatch using `PORTFOLIO_DISPATCH_TOKEN`, a fine-grained token limited to the site's repository. The first complete run, started by hand, recorded 99.9% across 14 modules at `da9b757` on 2026-10-04.
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
[![CI](https://img.shields.io/github/actions/workflow/status/jacksonfergusondev/protostar/ci.yml?color=22d3ee&labelColor=0A0A0A&label=CI)](https://github.com/jacksonfergusondev/protostar/actions/workflows/ci.yml)
[![Release](https://img.shields.io/github/actions/workflow/status/jacksonfergusondev/protostar/release.yml?color=22d3ee&labelColor=0A0A0A&label=release)](https://github.com/jacksonfergusondev/protostar/actions/workflows/release.yml)
[![Codecov](https://img.shields.io/codecov/c/github/JacksonFergusonDev/protostar?color=22d3ee&labelColor=0A0A0A&logo=codecov&logoColor=white)](https://codecov.io/gh/JacksonFergusonDev/protostar)
[![Engine mutation score](https://img.shields.io/endpoint?url=https%3A%2F%2Fprotostar.jacksonferguson.me%2Fbenchmarks%2Fmutation-latest.json&label=engine%20mutation%20score&color=22d3ee&labelColor=0A0A0A)](https://protostar.jacksonferguson.me/benchmarks/)
[![Engine mutation score](https://img.shields.io/endpoint?url=https%3A%2F%2Fprotostar.jacksonferguson.me%2Fmetrics%2Fmutation-latest.json&label=engine%20mutation%20score&color=22d3ee&labelColor=0A0A0A)](https://protostar.jacksonferguson.me/metrics/)
[![Python](https://img.shields.io/badge/python-3.12+-22d3ee?labelColor=0A0A0A&logo=python&logoColor=white)](https://www.python.org/downloads/)
[![Documentation](https://img.shields.io/badge/docs-gh--pages-22d3ee?labelColor=0A0A0A&logo=github&logoColor=white)](https://protostar.jacksonferguson.me/)
[![License](https://img.shields.io/badge/license-MIT-22d3ee?labelColor=0A0A0A)](LICENSE)
Expand Down Expand Up @@ -212,7 +212,7 @@ Protostar edits other people's work, so it's built to be careful:
- **The engine is separate from the interface.** The same core runs behind the interactive editor, the command line, and coding agents, which is why every command can also answer in JSON.
- **It's tested thoroughly.** Over 3,500 tests, including runs of the real tools, pass on Linux, macOS, and Windows, with strict type checking and at least 85% coverage.
- **Its docs are checked against its code.** Before every push, a script confirms that the commands, errors, exit codes, and file paths the docs mention still match the code.
- **Performance stays visible.** CI tracks help-command startup and the recipe editor's first frame, flagging large regressions; see the [CI performance history](https://protostar.jacksonferguson.me/benchmarks/).
- **Performance stays visible.** CI tracks help-command startup and the recipe editor's first frame, flagging large regressions; see the [metrics dashboard](https://protostar.jacksonferguson.me/metrics/).

---

Expand Down
Loading
Loading