Repository navigation
Description
Summary
Split the packaged CLI startup measurement out of the broader benchmark request github/squad#2476. This is a measurement-only issue for two commands: the built CLI's help and version entry points. It does not authorize startup optimization or a new benchmark framework.
Parent: github/squad#2481.
Proposed primary owner: cli-dev.
Existing evidence and blocker
At audited revision 47082a81799710c3e457b594123d190c00596ebf, scripts/measure-cold-start.mjs exists, but built CLI startup could not be measured. TypeScript was unavailable, locked dependency restoration returned a configured-feed E404 for hono-4.13.17.tgz, and dist was absent. There is no established startup latency or RSS result yet, and no startup bottleneck/speedup is claimed.
Use the existing built/shipping CLI entry point and runner after the documented restore/build succeeds. Source-loader or SDK-import probes are not substitutes, as illustrated by the packaged-path correction in microsoft/aspire#16704.
Scope
- In: help/version process-start-to-exit wall time and process maximum RSS, existing runner output/error handling, portable reproduction instructions and raw artifacts.
- Out: interactive shell/session resume, model calls, config/charter/memory/backend benchmarks, lazy-import changes, CI gates, cross-platform rollout and solving package-feed infrastructure in product code.
Measurable acceptance and tests
- For each command, collect three rounds of at least 20 fresh child processes on the same recorded SHA/build/environment. Record first observations separately and label OS filesystem-cache state; do not describe warm-filesystem fresh processes as cold storage.
- Publish every raw elapsed/RSS observation, exit status, exact command, SHA, Node/platform/build mode and per-round p50/p95. RSS must be labeled as process maximum RSS, not allocation volume or total agent-session memory.
- Run without inference/network workload or user-home mutation. Verify expected help/version output and successful exit on every measured invocation; keep output equivalent to the unmodified built CLI.
- Focused runner tests prove missing
dist, child nonzero exit, timeout and missing/invalid measurement data fail explicitly rather than emit a successful zero/NaN sample. Quantile/output-shape tests use deterministic inputs. - Publish a portable command and shared results artifact; no dependency on a private Copilot session path. If restore remains blocked, retain the exact blocker and leave measurement acceptance incomplete.
Done means
A reproducible, correctly labeled baseline for these two commands and failure-sensitive runner tests. No performance improvement is required or claimed by this measurement issue. Any identified startup optimization requires its own narrowly scoped, measured follow-up. The rollup github/squad#2481 tracks this result.
Activity