feat(benchmarks): pnpm workspace + port SDK + switch consumers#207
feat(benchmarks): pnpm workspace + port SDK + switch consumers#207HeyGarrison wants to merge 6 commits into
Conversation
HeyGarrison
commented
Jul 22, 2026
- fix(cli): make bundle:scale paths relative to packages/cli cwd (e8638bf)
- feat(bench): switch CLI consumers to benchmarks package + migrate CI to pnpm (2552191)
- feat: convert benchmarks to pnpm workspaces with packages/bench + packages/cli (32bf3e1)
Contributor License AgreementAll contributors are covered by a CLA. |
There was a problem hiding this comment.
Pull request overview
This PR converts the repo into a pnpm workspace with separate packages/bench (the benchmark orchestrator SDK) and packages/cli (the benchmark runner/consumer), then switches scale/storage/etc. CLI consumers from @computesdk/bench to the new local benchmarks workspace package and updates CI workflows accordingly.
Changes:
- Introduces pnpm workspace layout (
packages/bench,packages/cli) and migrates CI workflows to pnpm. - Ports the benchmark orchestrator client/worker helpers into
packages/benchand updates CLI code to import frombenchmarks. - Refactors benchmark runners (sandbox/storage/snapshot-fork/browser/scale) and adjusts result paths to the repo root
results/directory.
Reviewed changes
Copilot reviewed 44 out of 85 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| src/scale/Dockerfile | Removes old root-level scale Dockerfile (moved under packages/cli). |
| pnpm-workspace.yaml | Declares pnpm workspace packages (packages/bench, packages/cli). |
| packages/cli/tsconfig.json | Adds CLI TypeScript configuration for the new workspace package. |
| packages/cli/src/util/timeout.ts | Adds a shared promise timeout helper. |
| packages/cli/src/util/stats.ts | Adds shared percentile/stats helpers (used by sandbox benchmarks). |
| packages/cli/src/util/error.ts | Adds error formatting that preserves provider .cause chains. |
| packages/cli/src/update-readme.ts | Updates repo-root path resolution for new workspace layout. |
| packages/cli/src/storage/types.ts | Adds storage benchmark types and snapshot/fork override config shape. |
| packages/cli/src/storage/stats.ts | Adds shared rounding/stats helpers for storage benchmarks. |
| packages/cli/src/storage/snapshot-fork-types.ts | Introduces snapshot/fork dataset presets + result types. |
| packages/cli/src/storage/snapshot-fork-benchmark.ts | Implements snapshot/fork benchmark runner + scoring + JSON writer. |
| packages/cli/src/storage/scoring.ts | Adds composite scoring/sorting for storage benchmark results. |
| packages/cli/src/storage/providers.ts | Defines storage provider configs using StorageSDK adapters. |
| packages/cli/src/storage/generate-svg.ts | Updates repo-root path resolution for rendered storage SVGs. |
| packages/cli/src/storage/generate-snapshot-fork-svg.ts | Updates repo-root path resolution for rendered snapshot/fork SVGs. |
| packages/cli/src/storage/benchmark.ts | Implements upload/download/throughput storage benchmark + JSON writer. |
| packages/cli/src/scale/types.ts | Adds types for scale/burst-100k result shapes and metrics. |
| packages/cli/src/scale/sinks/tigris.ts | Adds Tigris S3-compatible sink for uploading raw/meta/metrics/logs. |
| packages/cli/src/scale/sdk-coordinator.ts | Switches orchestrator client imports to benchmarks workspace package. |
| packages/cli/src/scale/scripts/watch.ts | Switches orchestrator client imports to benchmarks. |
| packages/cli/src/scale/scripts/start.ts | Switches orchestrator client imports to benchmarks. |
| packages/cli/src/scale/scripts/spinup-idle.ts | Adds script to provision idle Namespace VMs for capacity dry-runs. |
| packages/cli/src/scale/scripts/kill.ts | Adds script to list/destroy live Namespace VMs for a run prefix. |
| packages/cli/src/scale/scripts/aggregate.ts | Switches orchestrator client imports to benchmarks. |
| packages/cli/src/scale/runner.ts | Adds two-phase burst lifecycle runner + structured result emission. |
| packages/cli/src/scale/providers.ts | Defines burst-100k provider participation and keep-alive settings. |
| packages/cli/src/scale/logger.ts | Adds structured logger with in-memory transcript for uploads. |
| packages/cli/src/scale/Dockerfile.norun | Adds a “bundle but don’t auto-run coordinator” scale image Dockerfile. |
| packages/cli/src/scale/Dockerfile | Adds pnpm-workspace-aware scale image Dockerfile that bundles coordinator. |
| packages/cli/src/scale/bench-reporter.ts | Switches orchestrator client import to benchmarks. |
| packages/cli/src/sandbox/types.ts | Adds sandbox benchmark types (TTI, modes, results). |
| packages/cli/src/sandbox/table.ts | Adds CLI table rendering + JSON writer for sandbox benchmark results. |
| packages/cli/src/sandbox/staggered.ts | Adds staggered-mode sandbox benchmark runner. |
| packages/cli/src/sandbox/scoring.ts | Adds sandbox composite scoring + sorting utilities. |
| packages/cli/src/sandbox/providers.ts | Rehomes sandbox provider list under packages/cli workspace. |
| packages/cli/src/sandbox/generate-svg.ts | Updates repo-root path resolution for sandbox SVG rendering. |
| packages/cli/src/sandbox/generate-pricing-svg.ts | Updates repo-root path resolution for pricing SVG rendering. |
| packages/cli/src/sandbox/dax.ts | Adds DAX workload benchmark runner using local dax-benchmark.sh. |
| packages/cli/src/sandbox/concurrent.ts | Adds concurrent/burst-style sandbox benchmark runner. |
| packages/cli/src/sandbox/benchmark.ts | Adds core sandbox benchmark iteration logic + reuse detection probe. |
| packages/cli/src/run.ts | Updates results output directories to repo root and wires new modes. |
| packages/cli/src/merge-results.ts | Updates repo-root path resolution for merging results under workspaces. |
| packages/cli/src/ingest.ts | Updates repo-root path resolution for ingest tooling under workspaces. |
| packages/cli/src/env.ts | Updates .env resolution to repo root under workspaces. |
| packages/cli/src/browser/types.ts | Adds browser benchmark types. |
| packages/cli/src/browser/throughput-types.ts | Adds browser throughput benchmark types/constants. |
| packages/cli/src/browser/throughput-scoring.ts | Adds throughput composite scoring + sorting. |
| packages/cli/src/browser/throughput-providers.ts | Adds throughput provider configs with standardized viewport options. |
| packages/cli/src/browser/scoring.ts | Adds browser benchmark composite scoring + sorting. |
| packages/cli/src/browser/providers.ts | Adds browser provider configs. |
| packages/cli/src/browser/generate-throughput-svg.ts | Updates repo-root path resolution for throughput SVG rendering. |
| packages/cli/src/browser/generate-svg.ts | Updates repo-root path resolution for browser SVG rendering. |
| packages/cli/src/browser/benchmark.ts | Adds browser benchmark runner using Playwright CDP connection. |
| packages/cli/scripts/find-northflank-plan.ts | Adds helper script to select a suitable Northflank deployment plan. |
| packages/cli/scripts/dax-benchmark.sh | Adds local bash script for DAX clone/install/typecheck workload. |
| packages/cli/scripts/build-e2b-template.ts | Adds helper to build an E2B template with fixed resources. |
| packages/cli/scripts/backfill.ts | Updates repo-root path resolution for backfill tooling under workspaces. |
| packages/cli/package.json | Adds CLI package scripts and depends on benchmarks workspace package. |
| packages/cli/env.example | Adds example environment variables for running benchmarks locally. |
| packages/bench/vitest.config.ts | Adds Vitest config for the benchmarks SDK package. |
| packages/bench/tsup.config.ts | Adds tsup build config for publishing benchmarks SDK. |
| packages/bench/tsconfig.json | Adds package tsconfig extending repo root config. |
| packages/bench/src/reporter.ts | Adds BenchmarkReporter wrapper for best-effort telemetry/reporting. |
| packages/bench/src/metrics.ts | Adds system metrics sampler for coordinators/workers. |
| packages/bench/src/index.ts | Exports client/worker helpers, reporter, metrics, and public types. |
| packages/bench/src/tests/integration.test.ts | Adds optional live integration smoke test (env-gated). |
| packages/bench/README.md | Adds usage/API docs for the new benchmarks SDK package. |
| packages/bench/package.json | Defines publishable benchmarks SDK package metadata and scripts. |
| package.json | Removes root scripts and drops @computesdk/bench dependency in favor of workspace package. |
| .npmrc | Adds pnpm setting (shamefully-hoist=true) for workspace installs. |
| .github/workflows/storage-benchmarks.yml | Migrates storage benchmark workflow to pnpm + new paths. |
| .github/workflows/snapshot-fork-benchmarks.yml | Migrates snapshot/fork workflow to pnpm + new paths. |
| .github/workflows/scale-image.yml | Updates scale image workflow paths to packages/cli/src/scale/**. |
| .github/workflows/sandbox-tti-benchmarks.yml | Migrates sandbox TTI workflow to pnpm + new paths. |
| .github/workflows/sandbox-dax-benchmarks.yml | Migrates DAX workflow to pnpm + new paths. |
| .github/workflows/pricing-svg.yml | Migrates pricing SVG workflow to pnpm. |
| .github/workflows/browser-throughput-benchmarks.yml | Migrates browser throughput workflow to pnpm + new paths. |
| .github/workflows/browser-benchmarks.yml | Migrates browser benchmark workflow to pnpm + new paths. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
100e6d7 to
b059020
Compare
…kages/cli Port the @computesdk/bench SDK verbatim into packages/bench (published name 'benchmarks', v0.2.0) and move the CLI runner tree into packages/cli (@benchmarks/cli, private). Add pnpm-workspace.yaml, replace package-lock.json with pnpm-lock.yaml, adjust __dirname/ROOT path math in packages/cli/src to resolve to the repo root, and move the @computesdk/bench dependency to packages/cli as benchmarks: workspace:*. results/ stays at repo root.
…to pnpm - Switch all 5 packages/cli/src/scale consumers from @computesdk/bench to benchmarks - Update packages/bench/README examples to the new package name - Add repo-root .npmrc (shamefully-hoist=true) so packages/cli resolves root provider pins + CLI tooling without changing provider version pins - Migrate 8 benchmark workflows npm->pnpm: pnpm/action-setup, cache: pnpm, pnpm install --frozen-lockfile, pnpm --filter @benchmarks/cli run, and repoint paths/run steps + git add to packages/cli/src and pnpm-lock.yaml - Point scale-image.yml + Dockerfile at packages/cli/src/scale/Dockerfile and make the Dockerfile pnpm-workspace aware - Fix bundle:scale outfile to packages/cli/dist/scale.cjs; add cli typecheck script + es2022 module so pnpm -r typecheck covers both packages
The bundle:scale script used repo-root-relative paths that only resolved from repo root, breaking 'pnpm --filter @benchmarks/cli run bundle:scale'. Use paths relative to the package cwd so it works via workspace filter; bundle:burst-100k delegates to it. The Dockerfile keeps its own inline repo-root esbuild command for backwards compatibility.
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
- Move src/ai-gateway/* to packages/cli/src/ai-gateway/* (master added this tree post-F2-cherry-pick) - Add 6 master-added bench:ai-gateway:* + generate-ai-gateway-svg scripts to packages/cli/package.json with paths rewritten to packages/cli/src/... - Empty root package.json scripts per F2 intent (bench:* / generate:* / bundle:* / update-readme all live in packages/cli/package.json now) - Adjust __dirname/.. path math in packages/cli/src/ai-gateway/generate-svg.ts (was '../..' at src/, becomes '../../..' one level deeper under packages/cli/) - Path-fix 62 cli scripts that referenced src/... instead of packages/cli/src/...
b059020 to
255d621
Compare
- Remove @computesdk/bench from root deps (validation contract expects it only in packages/cli/package.json as benchmarks: workspace:*) - Add benchmarks: workspace:* to packages/cli/package.json dependencies - Bump @computesdk/createos-sandbox 0.1.3 -> 0.1.4 (master updated) - Bump @computesdk/upstash 0.3.7 -> 0.3.9 (master updated) - Drop @changesets/cli from root devDeps (was lifted from packages/cli via shamefully-hoist)
| # - anchorbrowser | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - uses: pnpm/action-setup@v4 |
There was a problem hiding this comment.
P2: Third-party pnpm/action-setup action is not pinned to a commit SHA
Uses a mutable version tag for a third-party GitHub Action, allowing supply chain attacks if the tag is retargeted.
Pin the action to a specific commit SHA and include the version comment.
AI prompt
Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.
<file name=".github/workflows/browser-benchmarks.yml">
<violation number="1" location=".github/workflows/browser-benchmarks.yml:47">
<priority>P2</priority>
<title>Third-party pnpm/action-setup action is not pinned to a commit SHA</title>
<evidence>` - uses: pnpm/action-setup@v4` uses a mutable version tag instead of a pinned commit SHA. This action is a third-party action (not from the actions/ organization). A mutable tag can be retargeted by the publisher to point to malicious code without any change to this repository. The workflows using this action have access to repository secrets.</evidence>
<recommendation>Pin `pnpm/action-setup` to a specific commit SHA (e.g., `pnpm/action-setup@1234567... # v4.x.x`) to prevent supply chain attacks via tag retargeting. Apply this change to all occurrences across all workflow files. Refer to the action's releases page for the current v4 SHA.</recommendation>
</violation>
</file>
| - steel | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - uses: pnpm/action-setup@v4 |
There was a problem hiding this comment.
P2: Third-party pnpm/action-setup action is not pinned to a commit SHA
Uses a mutable version tag for a third-party GitHub Action, allowing supply chain attacks if the tag is retargeted.
Pin the action to a specific commit SHA and include the version comment.
AI prompt
Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.
<file name=".github/workflows/browser-throughput-benchmarks.yml">
<violation number="1" location=".github/workflows/browser-throughput-benchmarks.yml:98">
<priority>P2</priority>
<title>Third-party pnpm/action-setup action is not pinned to a commit SHA</title>
<evidence>` - uses: pnpm/action-setup@v4` uses a mutable version tag instead of a pinned commit SHA. This action is a third-party action (not from the actions/ organization). A mutable tag can be retargeted by the publisher to point to malicious code without any change to this repository. The workflows using this action have access to repository secrets.</evidence>
<recommendation>Pin `pnpm/action-setup` to a specific commit SHA (e.g., `pnpm/action-setup@1234567... # v4.x.x`) to prevent supply chain attacks via tag retargeting. Apply this change to all occurrences across all workflow files. Refer to the action's releases page for the current v4 SHA.</recommendation>
</violation>
</file>
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - uses: pnpm/action-setup@v4 |
There was a problem hiding this comment.
P2: Third-party pnpm/action-setup action is not pinned to a commit SHA
Uses a mutable version tag for a third-party GitHub Action, allowing supply chain attacks if the tag is retargeted.
Pin the action to a specific commit SHA and include the version comment.
AI prompt
Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.
<file name=".github/workflows/pricing-svg.yml">
<violation number="1" location=".github/workflows/pricing-svg.yml:19">
<priority>P2</priority>
<title>Third-party pnpm/action-setup action is not pinned to a commit SHA</title>
<evidence>` - uses: pnpm/action-setup@v4` uses a mutable version tag instead of a pinned commit SHA. This action is a third-party action (not from the actions/ organization). A mutable tag can be retargeted by the publisher to point to malicious code without any change to this repository. The workflows using this action have access to repository secrets.</evidence>
<recommendation>Pin `pnpm/action-setup` to a specific commit SHA (e.g., `pnpm/action-setup@1234567... # v4.x.x`) to prevent supply chain attacks via tag retargeting. Apply this change to all occurrences across all workflow files. Refer to the action's releases page for the current v4 SHA.</recommendation>
</violation>
</file>
| - vercel | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - uses: pnpm/action-setup@v4 |
There was a problem hiding this comment.
P2: Third-party pnpm/action-setup action is not pinned to a commit SHA
Uses a mutable version tag for a third-party GitHub Action, allowing supply chain attacks if the tag is retargeted.
Pin the action to a specific commit SHA and include the version comment.
AI prompt
Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.
<file name=".github/workflows/sandbox-dax-benchmarks.yml">
<violation number="1" location=".github/workflows/sandbox-dax-benchmarks.yml:73">
<priority>P2</priority>
<title>Third-party pnpm/action-setup action is not pinned to a commit SHA</title>
<evidence>` - uses: pnpm/action-setup@v4` uses a mutable version tag instead of a pinned commit SHA. This action is a third-party action (not from the actions/ organization). A mutable tag can be retargeted by the publisher to point to malicious code without any change to this repository. The workflows using this action have access to repository secrets.</evidence>
<recommendation>Pin `pnpm/action-setup` to a specific commit SHA (e.g., `pnpm/action-setup@1234567... # v4.x.x`) to prevent supply chain attacks via tag retargeting. Apply this change to all occurrences across all workflow files. Refer to the action's releases page for the current v4 SHA.</recommendation>
</violation>
</file>
| - vercel | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - uses: pnpm/action-setup@v4 |
There was a problem hiding this comment.
P2: Third-party pnpm/action-setup action is not pinned to a commit SHA
Uses a mutable version tag for a third-party GitHub Action, allowing supply chain attacks if the tag is retargeted.
Pin the action to a specific commit SHA and include the version comment.
AI prompt
Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.
<file name=".github/workflows/sandbox-tti-benchmarks.yml">
<violation number="1" location=".github/workflows/sandbox-tti-benchmarks.yml:89">
<priority>P2</priority>
<title>Third-party pnpm/action-setup action is not pinned to a commit SHA</title>
<evidence>` - uses: pnpm/action-setup@v4` uses a mutable version tag instead of a pinned commit SHA. This action is a third-party action (not from the actions/ organization). A mutable tag can be retargeted by the publisher to point to malicious code without any change to this repository. The workflows using this action have access to repository secrets.</evidence>
<recommendation>Pin `pnpm/action-setup` to a specific commit SHA (e.g., `pnpm/action-setup@1234567... # v4.x.x`) to prevent supply chain attacks via tag retargeting. Apply this change to all occurrences across all workflow files. Refer to the action's releases page for the current v4 SHA.</recommendation>
</violation>
</file>
| # - vercel-blob | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - uses: pnpm/action-setup@v4 |
There was a problem hiding this comment.
P2: Third-party pnpm/action-setup action is not pinned to a commit SHA
Uses a mutable version tag for a third-party GitHub Action, allowing supply chain attacks if the tag is retargeted.
Pin the action to a specific commit SHA and include the version comment.
AI prompt
Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.
<file name=".github/workflows/snapshot-fork-benchmarks.yml">
<violation number="1" location=".github/workflows/snapshot-fork-benchmarks.yml:63">
<priority>P2</priority>
<title>Third-party pnpm/action-setup action is not pinned to a commit SHA</title>
<evidence>` - uses: pnpm/action-setup@v4` uses a mutable version tag instead of a pinned commit SHA. This action is a third-party action (not from the actions/ organization). A mutable tag can be retargeted by the publisher to point to malicious code without any change to this repository. The workflows using this action have access to repository secrets.</evidence>
<recommendation>Pin `pnpm/action-setup` to a specific commit SHA (e.g., `pnpm/action-setup@1234567... # v4.x.x`) to prevent supply chain attacks via tag retargeting. Apply this change to all occurrences across all workflow files. Refer to the action's releases page for the current v4 SHA.</recommendation>
</violation>
</file>
| file_size: ${{ (github.event_name == 'workflow_dispatch' && github.event.inputs.file_size != '' && fromJson(format('["{0}"]', github.event.inputs.file_size))) || (github.event_name == 'push' && fromJson('["1MB"]')) || fromJson('["1MB","4MB","10MB","16MB"]') }} | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - uses: pnpm/action-setup@v4 |
There was a problem hiding this comment.
P2: Third-party pnpm/action-setup action is not pinned to a commit SHA
Uses a mutable version tag for a third-party GitHub Action, allowing supply chain attacks if the tag is retargeted.
Pin the action to a specific commit SHA and include the version comment.
AI prompt
Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.
<file name=".github/workflows/storage-benchmarks.yml">
<violation number="1" location=".github/workflows/storage-benchmarks.yml:62">
<priority>P2</priority>
<title>Third-party pnpm/action-setup action is not pinned to a commit SHA</title>
<evidence>` - uses: pnpm/action-setup@v4` uses a mutable version tag instead of a pinned commit SHA. This action is a third-party action (not from the actions/ organization). A mutable tag can be retargeted by the publisher to point to malicious code without any change to this repository. The workflows using this action have access to repository secrets.</evidence>
<recommendation>Pin `pnpm/action-setup` to a specific commit SHA (e.g., `pnpm/action-setup@1234567... # v4.x.x`) to prevent supply chain attacks via tag retargeting. Apply this change to all occurrences across all workflow files. Refer to the action's releases page for the current v4 SHA.</recommendation>
</violation>
</file>