Skip to content

perf(cli): establish built help and version startup baselines #2482

Backlog

Description

@spboyer

Summary

Split the packaged CLI startup measurement out of the broader benchmark request github/squad#2476. This is a measurement-only issue for two commands: the built CLI's help and version entry points. It does not authorize startup optimization or a new benchmark framework.

Parent: github/squad#2481.

Proposed primary owner: cli-dev.

Existing evidence and blocker

At audited revision 47082a81799710c3e457b594123d190c00596ebf, scripts/measure-cold-start.mjs exists, but built CLI startup could not be measured. TypeScript was unavailable, locked dependency restoration returned a configured-feed E404 for hono-4.13.17.tgz, and dist was absent. There is no established startup latency or RSS result yet, and no startup bottleneck/speedup is claimed.

Use the existing built/shipping CLI entry point and runner after the documented restore/build succeeds. Source-loader or SDK-import probes are not substitutes, as illustrated by the packaged-path correction in microsoft/aspire#16704.

Scope

  • In: help/version process-start-to-exit wall time and process maximum RSS, existing runner output/error handling, portable reproduction instructions and raw artifacts.
  • Out: interactive shell/session resume, model calls, config/charter/memory/backend benchmarks, lazy-import changes, CI gates, cross-platform rollout and solving package-feed infrastructure in product code.

Measurable acceptance and tests

  • For each command, collect three rounds of at least 20 fresh child processes on the same recorded SHA/build/environment. Record first observations separately and label OS filesystem-cache state; do not describe warm-filesystem fresh processes as cold storage.
  • Publish every raw elapsed/RSS observation, exit status, exact command, SHA, Node/platform/build mode and per-round p50/p95. RSS must be labeled as process maximum RSS, not allocation volume or total agent-session memory.
  • Run without inference/network workload or user-home mutation. Verify expected help/version output and successful exit on every measured invocation; keep output equivalent to the unmodified built CLI.
  • Focused runner tests prove missing dist, child nonzero exit, timeout and missing/invalid measurement data fail explicitly rather than emit a successful zero/NaN sample. Quantile/output-shape tests use deterministic inputs.
  • Publish a portable command and shared results artifact; no dependency on a private Copilot session path. If restore remains blocked, retain the exact blocker and leave measurement acceptance incomplete.

Done means

A reproducible, correctly labeled baseline for these two commands and failure-sensitive runner tests. No performance improvement is required or claimed by this measurement issue. Any identified startup optimization requires its own narrowly scoped, measured follow-up. The rollup github/squad#2481 tracks this result.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions