Skip to content

feat(benchmarks): pnpm workspace + port SDK + switch consumers#207

Open
HeyGarrison wants to merge 6 commits into
masterfrom
feat/bench-sdk-workspace
Open

feat(benchmarks): pnpm workspace + port SDK + switch consumers#207
HeyGarrison wants to merge 6 commits into
masterfrom
feat/bench-sdk-workspace

Conversation

@HeyGarrison

Copy link
Copy Markdown
Collaborator
  • fix(cli): make bundle:scale paths relative to packages/cli cwd (e8638bf)
  • feat(bench): switch CLI consumers to benchmarks package + migrate CI to pnpm (2552191)
  • feat: convert benchmarks to pnpm workspaces with packages/bench + packages/cli (32bf3e1)

@open-cla

open-cla Bot commented Jul 22, 2026

Copy link
Copy Markdown

Contributor License Agreement

All contributors are covered by a CLA.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 potential issues.

View 2 additional findings in Devin Review.

Open in Devin Review

Comment thread package.json
Comment thread .github/workflows/sandbox-dax-benchmarks.yml

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR converts the repo into a pnpm workspace with separate packages/bench (the benchmark orchestrator SDK) and packages/cli (the benchmark runner/consumer), then switches scale/storage/etc. CLI consumers from @computesdk/bench to the new local benchmarks workspace package and updates CI workflows accordingly.

Changes:

  • Introduces pnpm workspace layout (packages/bench, packages/cli) and migrates CI workflows to pnpm.
  • Ports the benchmark orchestrator client/worker helpers into packages/bench and updates CLI code to import from benchmarks.
  • Refactors benchmark runners (sandbox/storage/snapshot-fork/browser/scale) and adjusts result paths to the repo root results/ directory.

Reviewed changes

Copilot reviewed 44 out of 85 changed files in this pull request and generated no comments.

Show a summary per file
File Description
src/scale/Dockerfile Removes old root-level scale Dockerfile (moved under packages/cli).
pnpm-workspace.yaml Declares pnpm workspace packages (packages/bench, packages/cli).
packages/cli/tsconfig.json Adds CLI TypeScript configuration for the new workspace package.
packages/cli/src/util/timeout.ts Adds a shared promise timeout helper.
packages/cli/src/util/stats.ts Adds shared percentile/stats helpers (used by sandbox benchmarks).
packages/cli/src/util/error.ts Adds error formatting that preserves provider .cause chains.
packages/cli/src/update-readme.ts Updates repo-root path resolution for new workspace layout.
packages/cli/src/storage/types.ts Adds storage benchmark types and snapshot/fork override config shape.
packages/cli/src/storage/stats.ts Adds shared rounding/stats helpers for storage benchmarks.
packages/cli/src/storage/snapshot-fork-types.ts Introduces snapshot/fork dataset presets + result types.
packages/cli/src/storage/snapshot-fork-benchmark.ts Implements snapshot/fork benchmark runner + scoring + JSON writer.
packages/cli/src/storage/scoring.ts Adds composite scoring/sorting for storage benchmark results.
packages/cli/src/storage/providers.ts Defines storage provider configs using StorageSDK adapters.
packages/cli/src/storage/generate-svg.ts Updates repo-root path resolution for rendered storage SVGs.
packages/cli/src/storage/generate-snapshot-fork-svg.ts Updates repo-root path resolution for rendered snapshot/fork SVGs.
packages/cli/src/storage/benchmark.ts Implements upload/download/throughput storage benchmark + JSON writer.
packages/cli/src/scale/types.ts Adds types for scale/burst-100k result shapes and metrics.
packages/cli/src/scale/sinks/tigris.ts Adds Tigris S3-compatible sink for uploading raw/meta/metrics/logs.
packages/cli/src/scale/sdk-coordinator.ts Switches orchestrator client imports to benchmarks workspace package.
packages/cli/src/scale/scripts/watch.ts Switches orchestrator client imports to benchmarks.
packages/cli/src/scale/scripts/start.ts Switches orchestrator client imports to benchmarks.
packages/cli/src/scale/scripts/spinup-idle.ts Adds script to provision idle Namespace VMs for capacity dry-runs.
packages/cli/src/scale/scripts/kill.ts Adds script to list/destroy live Namespace VMs for a run prefix.
packages/cli/src/scale/scripts/aggregate.ts Switches orchestrator client imports to benchmarks.
packages/cli/src/scale/runner.ts Adds two-phase burst lifecycle runner + structured result emission.
packages/cli/src/scale/providers.ts Defines burst-100k provider participation and keep-alive settings.
packages/cli/src/scale/logger.ts Adds structured logger with in-memory transcript for uploads.
packages/cli/src/scale/Dockerfile.norun Adds a “bundle but don’t auto-run coordinator” scale image Dockerfile.
packages/cli/src/scale/Dockerfile Adds pnpm-workspace-aware scale image Dockerfile that bundles coordinator.
packages/cli/src/scale/bench-reporter.ts Switches orchestrator client import to benchmarks.
packages/cli/src/sandbox/types.ts Adds sandbox benchmark types (TTI, modes, results).
packages/cli/src/sandbox/table.ts Adds CLI table rendering + JSON writer for sandbox benchmark results.
packages/cli/src/sandbox/staggered.ts Adds staggered-mode sandbox benchmark runner.
packages/cli/src/sandbox/scoring.ts Adds sandbox composite scoring + sorting utilities.
packages/cli/src/sandbox/providers.ts Rehomes sandbox provider list under packages/cli workspace.
packages/cli/src/sandbox/generate-svg.ts Updates repo-root path resolution for sandbox SVG rendering.
packages/cli/src/sandbox/generate-pricing-svg.ts Updates repo-root path resolution for pricing SVG rendering.
packages/cli/src/sandbox/dax.ts Adds DAX workload benchmark runner using local dax-benchmark.sh.
packages/cli/src/sandbox/concurrent.ts Adds concurrent/burst-style sandbox benchmark runner.
packages/cli/src/sandbox/benchmark.ts Adds core sandbox benchmark iteration logic + reuse detection probe.
packages/cli/src/run.ts Updates results output directories to repo root and wires new modes.
packages/cli/src/merge-results.ts Updates repo-root path resolution for merging results under workspaces.
packages/cli/src/ingest.ts Updates repo-root path resolution for ingest tooling under workspaces.
packages/cli/src/env.ts Updates .env resolution to repo root under workspaces.
packages/cli/src/browser/types.ts Adds browser benchmark types.
packages/cli/src/browser/throughput-types.ts Adds browser throughput benchmark types/constants.
packages/cli/src/browser/throughput-scoring.ts Adds throughput composite scoring + sorting.
packages/cli/src/browser/throughput-providers.ts Adds throughput provider configs with standardized viewport options.
packages/cli/src/browser/scoring.ts Adds browser benchmark composite scoring + sorting.
packages/cli/src/browser/providers.ts Adds browser provider configs.
packages/cli/src/browser/generate-throughput-svg.ts Updates repo-root path resolution for throughput SVG rendering.
packages/cli/src/browser/generate-svg.ts Updates repo-root path resolution for browser SVG rendering.
packages/cli/src/browser/benchmark.ts Adds browser benchmark runner using Playwright CDP connection.
packages/cli/scripts/find-northflank-plan.ts Adds helper script to select a suitable Northflank deployment plan.
packages/cli/scripts/dax-benchmark.sh Adds local bash script for DAX clone/install/typecheck workload.
packages/cli/scripts/build-e2b-template.ts Adds helper to build an E2B template with fixed resources.
packages/cli/scripts/backfill.ts Updates repo-root path resolution for backfill tooling under workspaces.
packages/cli/package.json Adds CLI package scripts and depends on benchmarks workspace package.
packages/cli/env.example Adds example environment variables for running benchmarks locally.
packages/bench/vitest.config.ts Adds Vitest config for the benchmarks SDK package.
packages/bench/tsup.config.ts Adds tsup build config for publishing benchmarks SDK.
packages/bench/tsconfig.json Adds package tsconfig extending repo root config.
packages/bench/src/reporter.ts Adds BenchmarkReporter wrapper for best-effort telemetry/reporting.
packages/bench/src/metrics.ts Adds system metrics sampler for coordinators/workers.
packages/bench/src/index.ts Exports client/worker helpers, reporter, metrics, and public types.
packages/bench/src/tests/integration.test.ts Adds optional live integration smoke test (env-gated).
packages/bench/README.md Adds usage/API docs for the new benchmarks SDK package.
packages/bench/package.json Defines publishable benchmarks SDK package metadata and scripts.
package.json Removes root scripts and drops @computesdk/bench dependency in favor of workspace package.
.npmrc Adds pnpm setting (shamefully-hoist=true) for workspace installs.
.github/workflows/storage-benchmarks.yml Migrates storage benchmark workflow to pnpm + new paths.
.github/workflows/snapshot-fork-benchmarks.yml Migrates snapshot/fork workflow to pnpm + new paths.
.github/workflows/scale-image.yml Updates scale image workflow paths to packages/cli/src/scale/**.
.github/workflows/sandbox-tti-benchmarks.yml Migrates sandbox TTI workflow to pnpm + new paths.
.github/workflows/sandbox-dax-benchmarks.yml Migrates DAX workflow to pnpm + new paths.
.github/workflows/pricing-svg.yml Migrates pricing SVG workflow to pnpm.
.github/workflows/browser-throughput-benchmarks.yml Migrates browser throughput workflow to pnpm + new paths.
.github/workflows/browser-benchmarks.yml Migrates browser benchmark workflow to pnpm + new paths.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@HeyGarrison
HeyGarrison force-pushed the feat/bench-sdk-workspace branch from 100e6d7 to b059020 Compare July 22, 2026 20:54
HeyGarrison and others added 5 commits July 22, 2026 21:02
…kages/cli

Port the @computesdk/bench SDK verbatim into packages/bench (published name
'benchmarks', v0.2.0) and move the CLI runner tree into packages/cli
(@benchmarks/cli, private). Add pnpm-workspace.yaml, replace package-lock.json
with pnpm-lock.yaml, adjust __dirname/ROOT path math in packages/cli/src to
resolve to the repo root, and move the @computesdk/bench dependency to
packages/cli as benchmarks: workspace:*. results/ stays at repo root.
…to pnpm

- Switch all 5 packages/cli/src/scale consumers from @computesdk/bench to benchmarks
- Update packages/bench/README examples to the new package name
- Add repo-root .npmrc (shamefully-hoist=true) so packages/cli resolves root
  provider pins + CLI tooling without changing provider version pins
- Migrate 8 benchmark workflows npm->pnpm: pnpm/action-setup, cache: pnpm,
  pnpm install --frozen-lockfile, pnpm --filter @benchmarks/cli run, and
  repoint paths/run steps + git add to packages/cli/src and pnpm-lock.yaml
- Point scale-image.yml + Dockerfile at packages/cli/src/scale/Dockerfile and
  make the Dockerfile pnpm-workspace aware
- Fix bundle:scale outfile to packages/cli/dist/scale.cjs; add cli typecheck
  script + es2022 module so pnpm -r typecheck covers both packages
The bundle:scale script used repo-root-relative paths that only resolved
from repo root, breaking 'pnpm --filter @benchmarks/cli run bundle:scale'.
Use paths relative to the package cwd so it works via workspace filter;
bundle:burst-100k delegates to it. The Dockerfile keeps its own inline
repo-root esbuild command for backwards compatibility.
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
- Move src/ai-gateway/* to packages/cli/src/ai-gateway/* (master added this tree
  post-F2-cherry-pick)
- Add 6 master-added bench:ai-gateway:* + generate-ai-gateway-svg scripts to
  packages/cli/package.json with paths rewritten to packages/cli/src/...
- Empty root package.json scripts per F2 intent (bench:* / generate:* /
  bundle:* / update-readme all live in packages/cli/package.json now)
- Adjust __dirname/.. path math in packages/cli/src/ai-gateway/generate-svg.ts
  (was '../..' at src/, becomes '../../..' one level deeper under packages/cli/)
- Path-fix 62 cli scripts that referenced src/... instead of packages/cli/src/...
@HeyGarrison
HeyGarrison force-pushed the feat/bench-sdk-workspace branch from b059020 to 255d621 Compare July 22, 2026 21:05
- Remove @computesdk/bench from root deps (validation contract expects it
  only in packages/cli/package.json as benchmarks: workspace:*)
- Add benchmarks: workspace:* to packages/cli/package.json dependencies
- Bump @computesdk/createos-sandbox 0.1.3 -> 0.1.4 (master updated)
- Bump @computesdk/upstash 0.3.7 -> 0.3.9 (master updated)
- Drop @changesets/cli from root devDeps (was lifted from packages/cli via shamefully-hoist)

@superagent-security superagent-security Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superagent found 1 security concern(s).

# - anchorbrowser
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Third-party pnpm/action-setup action is not pinned to a commit SHA

Uses a mutable version tag for a third-party GitHub Action, allowing supply chain attacks if the tag is retargeted.

Pin the action to a specific commit SHA and include the version comment.

AI prompt
Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.

<file name=".github/workflows/browser-benchmarks.yml">
<violation number="1" location=".github/workflows/browser-benchmarks.yml:47">
<priority>P2</priority>
<title>Third-party pnpm/action-setup action is not pinned to a commit SHA</title>
<evidence>`      - uses: pnpm/action-setup@v4` uses a mutable version tag instead of a pinned commit SHA. This action is a third-party action (not from the actions/ organization). A mutable tag can be retargeted by the publisher to point to malicious code without any change to this repository. The workflows using this action have access to repository secrets.</evidence>
<recommendation>Pin `pnpm/action-setup` to a specific commit SHA (e.g., `pnpm/action-setup@1234567... # v4.x.x`) to prevent supply chain attacks via tag retargeting. Apply this change to all occurrences across all workflow files. Refer to the action's releases page for the current v4 SHA.</recommendation>
</violation>
</file>

- steel
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Third-party pnpm/action-setup action is not pinned to a commit SHA

Uses a mutable version tag for a third-party GitHub Action, allowing supply chain attacks if the tag is retargeted.

Pin the action to a specific commit SHA and include the version comment.

AI prompt
Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.

<file name=".github/workflows/browser-throughput-benchmarks.yml">
<violation number="1" location=".github/workflows/browser-throughput-benchmarks.yml:98">
<priority>P2</priority>
<title>Third-party pnpm/action-setup action is not pinned to a commit SHA</title>
<evidence>`      - uses: pnpm/action-setup@v4` uses a mutable version tag instead of a pinned commit SHA. This action is a third-party action (not from the actions/ organization). A mutable tag can be retargeted by the publisher to point to malicious code without any change to this repository. The workflows using this action have access to repository secrets.</evidence>
<recommendation>Pin `pnpm/action-setup` to a specific commit SHA (e.g., `pnpm/action-setup@1234567... # v4.x.x`) to prevent supply chain attacks via tag retargeting. Apply this change to all occurrences across all workflow files. Refer to the action's releases page for the current v4 SHA.</recommendation>
</violation>
</file>

runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Third-party pnpm/action-setup action is not pinned to a commit SHA

Uses a mutable version tag for a third-party GitHub Action, allowing supply chain attacks if the tag is retargeted.

Pin the action to a specific commit SHA and include the version comment.

AI prompt
Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.

<file name=".github/workflows/pricing-svg.yml">
<violation number="1" location=".github/workflows/pricing-svg.yml:19">
<priority>P2</priority>
<title>Third-party pnpm/action-setup action is not pinned to a commit SHA</title>
<evidence>`      - uses: pnpm/action-setup@v4` uses a mutable version tag instead of a pinned commit SHA. This action is a third-party action (not from the actions/ organization). A mutable tag can be retargeted by the publisher to point to malicious code without any change to this repository. The workflows using this action have access to repository secrets.</evidence>
<recommendation>Pin `pnpm/action-setup` to a specific commit SHA (e.g., `pnpm/action-setup@1234567... # v4.x.x`) to prevent supply chain attacks via tag retargeting. Apply this change to all occurrences across all workflow files. Refer to the action's releases page for the current v4 SHA.</recommendation>
</violation>
</file>

- vercel
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Third-party pnpm/action-setup action is not pinned to a commit SHA

Uses a mutable version tag for a third-party GitHub Action, allowing supply chain attacks if the tag is retargeted.

Pin the action to a specific commit SHA and include the version comment.

AI prompt
Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.

<file name=".github/workflows/sandbox-dax-benchmarks.yml">
<violation number="1" location=".github/workflows/sandbox-dax-benchmarks.yml:73">
<priority>P2</priority>
<title>Third-party pnpm/action-setup action is not pinned to a commit SHA</title>
<evidence>`      - uses: pnpm/action-setup@v4` uses a mutable version tag instead of a pinned commit SHA. This action is a third-party action (not from the actions/ organization). A mutable tag can be retargeted by the publisher to point to malicious code without any change to this repository. The workflows using this action have access to repository secrets.</evidence>
<recommendation>Pin `pnpm/action-setup` to a specific commit SHA (e.g., `pnpm/action-setup@1234567... # v4.x.x`) to prevent supply chain attacks via tag retargeting. Apply this change to all occurrences across all workflow files. Refer to the action's releases page for the current v4 SHA.</recommendation>
</violation>
</file>

- vercel
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Third-party pnpm/action-setup action is not pinned to a commit SHA

Uses a mutable version tag for a third-party GitHub Action, allowing supply chain attacks if the tag is retargeted.

Pin the action to a specific commit SHA and include the version comment.

AI prompt
Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.

<file name=".github/workflows/sandbox-tti-benchmarks.yml">
<violation number="1" location=".github/workflows/sandbox-tti-benchmarks.yml:89">
<priority>P2</priority>
<title>Third-party pnpm/action-setup action is not pinned to a commit SHA</title>
<evidence>`      - uses: pnpm/action-setup@v4` uses a mutable version tag instead of a pinned commit SHA. This action is a third-party action (not from the actions/ organization). A mutable tag can be retargeted by the publisher to point to malicious code without any change to this repository. The workflows using this action have access to repository secrets.</evidence>
<recommendation>Pin `pnpm/action-setup` to a specific commit SHA (e.g., `pnpm/action-setup@1234567... # v4.x.x`) to prevent supply chain attacks via tag retargeting. Apply this change to all occurrences across all workflow files. Refer to the action's releases page for the current v4 SHA.</recommendation>
</violation>
</file>

# - vercel-blob
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Third-party pnpm/action-setup action is not pinned to a commit SHA

Uses a mutable version tag for a third-party GitHub Action, allowing supply chain attacks if the tag is retargeted.

Pin the action to a specific commit SHA and include the version comment.

AI prompt
Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.

<file name=".github/workflows/snapshot-fork-benchmarks.yml">
<violation number="1" location=".github/workflows/snapshot-fork-benchmarks.yml:63">
<priority>P2</priority>
<title>Third-party pnpm/action-setup action is not pinned to a commit SHA</title>
<evidence>`      - uses: pnpm/action-setup@v4` uses a mutable version tag instead of a pinned commit SHA. This action is a third-party action (not from the actions/ organization). A mutable tag can be retargeted by the publisher to point to malicious code without any change to this repository. The workflows using this action have access to repository secrets.</evidence>
<recommendation>Pin `pnpm/action-setup` to a specific commit SHA (e.g., `pnpm/action-setup@1234567... # v4.x.x`) to prevent supply chain attacks via tag retargeting. Apply this change to all occurrences across all workflow files. Refer to the action's releases page for the current v4 SHA.</recommendation>
</violation>
</file>

file_size: ${{ (github.event_name == 'workflow_dispatch' && github.event.inputs.file_size != '' && fromJson(format('["{0}"]', github.event.inputs.file_size))) || (github.event_name == 'push' && fromJson('["1MB"]')) || fromJson('["1MB","4MB","10MB","16MB"]') }}
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Third-party pnpm/action-setup action is not pinned to a commit SHA

Uses a mutable version tag for a third-party GitHub Action, allowing supply chain attacks if the tag is retargeted.

Pin the action to a specific commit SHA and include the version comment.

AI prompt
Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.

<file name=".github/workflows/storage-benchmarks.yml">
<violation number="1" location=".github/workflows/storage-benchmarks.yml:62">
<priority>P2</priority>
<title>Third-party pnpm/action-setup action is not pinned to a commit SHA</title>
<evidence>`      - uses: pnpm/action-setup@v4` uses a mutable version tag instead of a pinned commit SHA. This action is a third-party action (not from the actions/ organization). A mutable tag can be retargeted by the publisher to point to malicious code without any change to this repository. The workflows using this action have access to repository secrets.</evidence>
<recommendation>Pin `pnpm/action-setup` to a specific commit SHA (e.g., `pnpm/action-setup@1234567... # v4.x.x`) to prevent supply chain attacks via tag retargeting. Apply this change to all occurrences across all workflow files. Refer to the action's releases page for the current v4 SHA.</recommendation>
</violation>
</file>

@superagent-security superagent-security Bot added the pr:flagged PR flagged for review by security analysis. label Jul 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr:flagged PR flagged for review by security analysis.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants