Add GitHub benchmark workflow - #412
Conversation
Adds a workflow-dispatch GitHub Actions job for the local demo benchmark harness. The job runs the standard LingBot and Omnidreams baseline/candidate benchmark suite, summarizes performance and quality metrics in the Actions summary, and uploads the generated benchmark artifacts for inspection.
Greptile SummaryThe PR adds an automated GPU benchmark workflow for the seeded OmniDreams quality scenario.
Confidence Score: 4/5The PR is not yet safe to merge because the benchmark summary remains incorrect and mutable external Actions can replace the reviewed runner code. The workflow still looks up Files Needing Attention: .github/workflows/local-demo-benchmarks.yml Important Files Changed
Reviews (5): Last reviewed commit: "Rename demo benchmark job and probe base..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| else (((. * 10 | round) / 10 | tostring) + "s") | ||
| end; | ||
| .scenarios[] | ||
| | "| \(.id) | \(.status) | \(.wall_time_s | duration) | \(metric("gen_fps") | fmt) | \(metric("quality_score") | fmt) | \(metric("quality_similarity_score") | fmt) | \(metric("quality_psnr_db") | fmt) | \(metric("quality_rmse") | fmt) |" |
There was a problem hiding this comment.
When the benchmark summary reads a completed scenario, it queries gen_fps, while the harness records FPS under normalized keys such as pixel_fps_median_fps or wall_present_fps, causing the “Median gen FPS” column to report n/a even when FPS data was collected.
Knowledge Base Used: Local Benchmark Harness
| echo "Detected GPU compute capability: ${compute_cap} -> sm_${arch}" | ||
|
|
||
| - name: Checkout | ||
| uses: actions/checkout@v4 |
There was a problem hiding this comment.
The workflow selects checkout, setup-proxy-cache, setup-uv, and upload-artifact through mutable tags or branches, allowing an upstream reference change to replace reviewed runner code and expose the checked-out repository or benchmark environment. Pin each action to a reviewed full commit SHA.
How this was verified: Each new uses: declaration resolves a tag or branch rather than a full immutable commit SHA.
Converts the local demo benchmark workflow from manual-only to an automatic, non-blocking PR and merge-queue canary. The workflow now runs the shorter 30-second LingBot and Omnidreams seeded quality scenarios, keeps baseline/candidate comparison for quality metrics, and still uploads the benchmark artifacts for review.
jarcherNV
changed the title
Narrow LingBot's pipeline encoder config to the concrete camera-control encoder it already requires so Tyro can parse the full runner schema used by the local benchmark canary. Add a smoke test for the benchmark runner arguments to catch future CLI schema regressions before they reach the benchmark workflow.
Remove LingBot from the automatic benchmark canary for now because the GitHub runner cache volume does not satisfy LingBot's 200 GiB Hugging Face checkpoint preflight. Revert the LingBot CLI typing/test changes from the previous patch and keep LingBot benchmark scenarios available for local/manual runs.
Rename the local demo benchmark workflow/check to Demo Benchmarks to avoid confusion with the existing Omnidreams WorldLens canary. Trim large uv cache directories before Actions cache upload, matching the existing GPU canary pattern, and add a temporary non-failing DNS/TCP reachability probe for the baseline artifact host.
1 participant
Adds a workflow-dispatch GitHub Actions job for the local demo benchmark harness. The job runs the standard LingBot and Omnidreams baseline/candidate benchmark suite, summarizes performance and quality metrics in the Actions summary, and uploads the generated benchmark artifacts for inspection.