Add per-project CI: build + Palomar comparator check (changed projects only) - #19
Merged
Conversation
…ing (full run via workflow_dispatch) Also allow a larger runner for the verify job via the LEAN_CI_RUNNER repository/organization variable (falls back to ubuntu-latest), and on a push with no usable before-commit diff against the merge-base with main instead of selecting every project.
Pull-request runs restore the most recent .lake/build for the same toolchain, manifest and lakefile and save the new one, so Lake rebuilds only changed modules; pushes to main and workflow_dispatch runs neither restore nor save. .lake/packages is not cached (Mathlib comes from lake exe cache get).
rfurman
approved these changes
Aug 28, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #18 (base branch
move-zeta23-subfolder); GitHub retargets this PR tomainautomatically once #18 merges.What it does
Adds
.github/workflows/lean-projects.yml(+.github/scripts/comparator-check.sh) so that every project subfolder is held to the Palomar registry's acceptance bar, and only projects whose files changed are re-verified.pull_request(any branch),pushtomain,workflow_dispatch(inputprojects, defaultall).detectjob (ubuntu-latest, seconds): projects = top-level directories containinglakefile.tomlorlakefile.lean; selected = exactly those with a file ingit diff --name-only base...head(PR:base.sha...head.sha; push:before...after). Changes outside the project directories — CI configuration under.github/**,README.md,LICENSE,.gitignore— select nothing, so a CI/doc-only change runs no verification (verifyis skipped and the run is green). A full re-verify of every project is a manualworkflow_dispatchrun withprojects = all(the default; a space-separated list of project names is also accepted). On a push with no usablebeforecommit (first push of a branch, or abeforeno longer in the history) the diff is taken against the merge-base withmain; if even that is impossible, nothing is selected and a warning points at the manual full run. Emits a JSON matrix.verify (project)job (matrix,fail-fast: false,runs-on: ${{ vars.LEAN_CI_RUNNER || 'ubuntu-latest' }}— set the repository/organization variableLEAN_CI_RUNNERto a larger-runner label such asubuntu-latest-16-coresto shorten builds; Lake already uses every core —timeout-minutes: 360, working directory = the project):--default-toolchain none); the toolchain comes from the project'slean-toolchain.lake exe cache getiflake-manifest.jsonlistsmathlib. On pull requests only,actions/cachethen restores the project's most recent.lake/buildfor the same toolchain /lake-manifest.json/ lakefile (keylakebuild-<os>-<project>-<hash>-<sha>, restore-key without the sha), so Lake rebuilds only the modules that changed, and saves the new build; pushes tomainandworkflow_dispatchruns build from scratch (no restore, no save)..lake/packagesis never cached.lake build(default targets); fails on anydeclaration uses 'sorry'outside the trusted comparator statement filesChallenge.lean/Challenge/<Topic>.lean.lake env comparator <config>for every configuration the project declares (comparator*.jsonnext to the lakefile, orcomparator/config*.json— forzeta23that iscomparator/config.json(17 statements) andcomparator/config-xiprime.json(6)). Each run must exit 0 and printYour solution is okay!. As in the registry,permitted_axiomsmust be withinpropext/Quot.sound/Classical.choice, and comparator is handed a copy of the config withenable_nanodaforced totrue, so the independent nanoda kernel re-check always runs. Comparator sandboxes itslake build Challenge|Solution,lean4exportandnanodainvocations with landrun (real landrun, built from the pinned commit, as the registry does on the same hosted runners; the optionalsystemd-runAF_UNIX guard from the comparator README is not applied).logs-<project>artifact.~/.elanand the tool builds (~/.cache/lean-ci-tools) are cached withactions/cache, keyed on the toolchain and the tool pins. Only first-party actions (actions/checkout,actions/cache,actions/upload-artifact), pinned by commit.permissions: contents: read;concurrency: ${{ github.workflow }}-${{ github.ref }}with cancel-in-progress.The Palomar bar and the tool pins
"Passes the Palomar bar" = the project builds on its pinned toolchain and
leanprover/comparator— statement equality of everySolutiontheorem with its Mathlib-onlyChallengenamesake, axiom audit, replay in the Lean kernel and in the independentnanodakernel vialean4export, insidelandrun— printsYour solution is okay!.Pins are copied from the registry's own verifier, PalomarRegistry/PalomarSubmission
.github/workflows/submission.yml(at commite215b18):v4.2.3, tarball sha256df0b2b3a…a8b2575674928e239f5bc452aab72d1dd7b0f132649468d5ca9db226849b41a6fff59d796ff19d0a8840(cargo build --release --locked)811cfff51ceaf3d9843708aa6d22e9b84ccac8b4(CGO_ENABLED=0 go install)zeta23:v4.33.0-rc2→9fb131bb100eb32ccf6836f14e4f8328d13b6792), and itslean-toolchainmust equal the project'sverify_submission.pyresolve_release_commitThe registry's verifier additionally wraps everything in an outer landrun boundary with confinement probes, a trusted-cache provenance audit and a canonical Challenge compile; those defend the registry against hostile submissions and are not reproduced here — this workflow reproduces the mechanical check (same tools, same pins, same success criterion), which is what a project must pass.
Adding a project
A new top-level directory with
lakefile.toml/lakefile.lean,lean-toolchain,lake-manifest.json, a Mathlib-onlyChallengemodule, aSolutionmodule, and at least one comparator config (comparator.jsonorcomparator/config*.json), plus a row in the root README table (the README gains a short "CI" section describing this). Nothing in the workflow names a project.Known limits / things to decide
zeta23is ~100k lines of Lean; on a standard 4-vCPUubuntu-latestrunner the coldlake buildplus two comparator runs (each exports and replays the full proof cone in two kernels) may take several hours. The job timeout is the hosted maximum (360 min). If this proves too slow or memory-bound, set theLEAN_CI_RUNNERvariable to a larger-runner label available to this repository — no workflow change needed.pinsstep..lakeis not cached (Mathlib comes fromlake exe cache get; the project itself is rebuilt from source each run, which is the point of the check).cd <project> && bash ../.github/scripts/comparator-check.sh(needs elan, git, go, cargo, jq; Linux for landrun).This PR touches only
.github/**,README.mdand.gitignore, so under the rule above its own run selects[]andverifyis skipped; the end-to-end exercise ofzeta23(detect → build → both comparator configs) is the manualworkflow_dispatchrun withprojects = all, or the first PR that toucheszeta23/.