test: add HTTP request regression snapshots by penalosa · Pull Request #290 · cloudflare/cf

penalosa · 2026-10-13T10:11:57Z

Adds a dedicated Vitest + MSW regression suite that runs the real cf CLI for every generated API command and snapshots its HTTP requests. Each command must reach its declared HTTP method and endpoint before its baseline can be accepted.

The current catalogue covers 2,952 generated commands and 1,169 command groups across 69,280 argument cases. Cases cover no arguments, required values, individual flags, enum and boolean values, pairwise flag presence, positional alternatives, command and option aliases, global options, local mode, body inputs, and uploads. Snapshots preserve method, URL, headers, JSON/binary bodies, and ordered multipart parts, with deterministic normalization for runtime and fixture details.

All commands listed in hand-written-commands.json and their exclusive groups remain excluded: local projects, external tools, and multi-request workflows need separate fixtures outside this suite's scope. Mock responses are empty success envelopes apart from zone-name lookup. Local request rewriting stays real, with Miniflare dispatch routed through MSW.

The regression task regenerates commands before testing. Twelve balanced test files run in twelve Linux GitHub Actions jobs; the small harness suite runs with the normal cross-platform CLI tests. Snapshots are marked as generated, and obsolete baselines have been removed after rebasing onto current main.

Validation:

pnpm check
pnpm --filter cf test:regression --maxWorkers=4 — 4,121 tests
Focused harness tests — 12 tests
Audited all 4,121 snapshot files for catalogue coverage, endpoint reachability, origins, and obsolete baselines

Test and CI changes only; no changeset is required.

pkg-pr-new · 2026-10-13T10:12:47Z

npm i https://pkg.pr.new/cf@290

commit: 942d6a7

github-actions · 2026-10-13T10:13:34Z

`cf --help` startup

	mean	stddev	min	max
PR	228.1 ms	2.6 ms	224.2 ms	233.1 ms
base	230.6 ms	3.8 ms	225.5 ms	237.6 ms

Δ (PR vs base): ↓ -2.5 ms (-1.1%)

Measured with hyperfine on ubuntu-latest (3 warmup runs, 20 timed runs). CI numbers carry ±a few ms of runner jitter; treat small deltas as noise.

ask-bonk · 2026-10-13T10:17:43Z

🤖 I've done a thorough review of your PR.

Adds a parallel MSW/Vitest snapshot suite that records request behavior for the generated and hand-written CLI command surface.

Warning: cf tunnels run is hand-written and bypasses the generic generated-command endpoint assertion. It performs /cfd_tunnel lookup requests, but is absent from workflowRequestPaths, so that request can silently disappear when its snapshot is updated.

"cf tunnels run": "/cfd_tunnel",
"cf tunnels tail": "/management",

Warning: The workflow assertion only checks a path substring, so it accepts prerequisite reads in place of the primary mutation. Preview deploy's GET .../deployments/latest passes in place of its POST .../deployments, and image deletion's HEAD passes in place of DELETE.

const workflowPath = workflowRequestPaths[command.command];
const workflowMethod =
  command.command === "cf previews deploy"
    ? "POST"
    : command.command === "cf containers images delete"
      ? "DELETE"
      : undefined;
if (workflowPath) {
  expect(
    recordings.some((recording) =>
      recording.requests.some(
        (request) =>
          (!workflowMethod || request.method === workflowMethod) &&
          new URL(request.url).pathname.includes(workflowPath)
      )
    ),
    `${command.command} never reached ${workflowPath}. Fix its fixture before updating the request baseline.`
  ).toBe(true);
}

ask-bonk · 2026-10-13T10:17:48Z

🤖 I've done a thorough review of your PR.

Adds HTTP request regression snapshots across the CLI surface.

Warning: cf tunnels run lacks workflow endpoint coverage.
Warning: Path-only workflow checks can accept preflight requests instead of mutations.

github run

ask-bonk · 2026-10-13T10:26:20Z

The workflow request guard has two gaps that let required hand-written API calls disappear when baselines are updated.

cf tunnels run is hand-written and bypasses the generated-command endpoint assertion. When no token is supplied it resolves the tunnel through GET /cfd_tunnel, but workflowRequestPaths does not require that request. Store an expected method with each path and include this flow:

export const workflowRequestPaths: Record<
	string,
	{ path: string; method: string }
> = {
	"cf ai run": { path: "/ai/run", method: "POST" },
	"cf docs search": { path: "/api/ai-search/search", method: "POST" },
	"cf registrar registrations create": {
		path: "/registrar/registrations",
		method: "POST",
	},
	"cf workers versions create": { path: "/versions", method: "POST" },
	"cf workers triggers deploy": { path: "/schedules", method: "PUT" },
	"cf previews deploy": { path: "/deployments", method: "POST" },
	"cf containers images list": { path: "/_catalog", method: "GET" },
	"cf containers images delete": { path: "/manifests/", method: "DELETE" },
	"cf d1 migrations list": { path: "/query", method: "POST" },
	"cf d1 migrations apply": { path: "/query", method: "POST" },
	"cf tunnels run": { path: "/cfd_tunnel", method: "GET" },
	"cf tunnels tail": { path: "/management", method: "POST" },
};

The suite checks only a pathname. Preview deploy records GET .../deployments/latest before its required POST .../deployments, and image deletion similarly sends HEAD before DELETE; dropping either mutation still satisfies the check. Require the mapped HTTP verb:

				const workflow = workflowRequestPaths[command.command];
				if (workflow) {
					expect(
						recordings.some((recording) =>
							recording.requests.some(
								(request) =>
									request.method === workflow.method &&
									new URL(request.url).pathname.includes(workflow.path)
							)
						),
						`${command.command} never reached ${workflow.method} ${workflow.path}. Fix its fixture before updating the request baseline.`
					).toBe(true);
				}

ask-bonk · 2026-10-13T10:26:48Z

Review: 2 findings.

🤖 I've done a thorough review of your PR.

Adds HTTP request regression snapshots and CI coverage.

Warning: cf tunnels run lacks a required API endpoint guard.
Warning: workflow guards check paths but not HTTP methods, allowing prerequisite requests to mask missing mutations.

Posted both fixes as GitHub suggestion comments.

github run

ask-bonk · 2026-10-13T10:30:59Z

The workflow guard needs to check the expected HTTP method as well as the path. The preview snapshot records GET .../deployments/latest before the required POST .../deployments, and image deletion records HEAD .../manifests/... before DELETE; removing a mutation and updating its snapshot would still pass.

export const workflowRequestPaths: Record<
	string,
	{ path: string; method: string }
> = {
	"cf ai run": { path: "/ai/run", method: "POST" },
	"cf docs search": { path: "/api/ai-search/search", method: "POST" },
	"cf registrar registrations create": {
		path: "/registrar/registrations",
		method: "POST",
	},
	"cf workers versions create": { path: "/versions", method: "POST" },
	"cf workers triggers deploy": { path: "/schedules", method: "PUT" },
	"cf previews deploy": { path: "/deployments", method: "POST" },
	"cf containers images list": { path: "/_catalog", method: "GET" },
	"cf containers images delete": { path: "/manifests/", method: "DELETE" },
	"cf d1 migrations list": { path: "/query", method: "POST" },
	"cf d1 migrations apply": { path: "/query", method: "POST" },
};

const workflow = workflowRequestPaths[command.command];
if (workflow) {
	expect(
		recordings.some((recording) =>
			recording.requests.some(
				(request) =>
					request.method === workflow.method &&
					new URL(request.url).pathname.includes(workflow.path)
			)
		),
		`${command.command} never reached ${workflow.method} ${workflow.path}. Fix its fixture before updating the request baseline.`
	).toBe(true);
}

ask-bonk · 2026-10-13T10:31:04Z

🤖 I've done a thorough review of your PR.

Adds parallel MSW request snapshots and Linux CI coverage.

Warning: Workflow guards accept prerequisite reads instead of required mutations. Posted a suggestion comment: test: add HTTP request regression snapshots #290 (comment)

github run

ask-bonk · 2026-10-13T10:35:00Z

No new concerns since the last review

github run

ask-bonk

The suite claims alias coverage, but every included invocation uses canonical option names. In particular, cf containers images delete is exercised with --force but never its public -f alias, so a regression in that yargs alias or telemetry alias mapping can be accepted by updating this snapshot. Generate one equivalent case for each OptionMeta.alias (all alias forms are represented, and the current included surface has -f).

ask-bonk · 2026-10-13T11:23:32Z

🤖 I've done a thorough review of your PR.

Adds HTTP request regression snapshots and CI coverage.

Warning: option aliases are not exercised. Posted a suggestion for alias cases, including cf containers images delete -f.

github run

ask-bonk · 2026-10-13T11:37:58Z

No new concerns since the last review

github run

ask-bonk · 2026-10-13T12:03:13Z

LGTM

github run

ask-bonk · 2026-11-04T14:04:12Z

LGTM!

github run

ask-bonk · 2026-11-04T14:11:09Z

🤖 I've done a thorough review of your PR.

Adds generated-command HTTP request snapshots and sharded CI coverage.

Warning: Global short aliases -q, -z, and -m lack regression cases. Posted an inline suggestion in cases.ts.

github run

ask-bonk · 2026-11-04T14:24:28Z

LGTM!

github run

petebacondarwin · 2026-11-04T21:04:04Z

I am not convinced that these additional tests add much value on top of the effective snapshot which is the generated code. Do we really need to run 69,000 tests on every change?

ask-bonk Bot reviewed Oct 13, 2026

View reviewed changes

Comment thread packages/cli/src/__tests__/regression/cases.ts Resolved

petebacondarwin requested review from a team and NuroDev and removed request for a team October 23, 2026 12:59

hajikelu mentioned this pull request Oct 31, 2026

ai to-markdown never sends a files file part, so every conversion fails with 6006 #463

Closed

penalosa and others added 12 commits November 4, 2026 13:52

test: add parallel HTTP regression snapshots for every cf command

5c2fc4f

test: exclude workflow commands from HTTP regression snapshots

4c5de0d

ci: run HTTP regression snapshots on Linux only

ea00d6a

chore: mark regression snapshots as generated

cb20965

test: simplify HTTP regression boundaries

72bac40

ci: run HTTP regression snapshots in a separate job

b9a8b8f

test: exclude cloudflared commands from HTTP regression snapshots

36099f7

ci: run HTTP regression snapshots in sharded workflow

5982ad6

test: use Vitest default workers for regression suite

4ce66d9

test: limit HTTP regression snapshots to generated commands

adaa05f

test: omit SDK runtime headers from regression snapshots

fddb78a

test: refresh HTTP regression coverage for current commands

a4127eb

petebacondarwin force-pushed the penalosa/regression-testing branch from 26d71fc to a4127eb Compare November 4, 2026 14:02

github-advanced-security Bot found potential problems Nov 4, 2026

View reviewed changes

Comment thread .github/workflows/regression.yml Fixed

ci: use self-repository action syntax for regression tests

a8fb7e4

ask-bonk Bot reviewed Nov 4, 2026

View reviewed changes

Comment thread packages/cli/src/__tests__/regression/cases.ts Resolved

test: cover global option aliases in HTTP snapshots

942d6a7

petebacondarwin marked this pull request as ready for review November 4, 2026 14:27

petebacondarwin marked this pull request as draft November 4, 2026 21:03

Conversation

penalosa commented Oct 13, 2026 • edited by petebacondarwin Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pkg-pr-new Bot commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

cf --help startup

Uh oh!

ask-bonk Bot commented Oct 13, 2026

Uh oh!

ask-bonk Bot commented Oct 13, 2026

Uh oh!

ask-bonk Bot commented Oct 13, 2026

Uh oh!

ask-bonk Bot commented Oct 13, 2026

Uh oh!

ask-bonk Bot commented Oct 13, 2026

Uh oh!

ask-bonk Bot commented Oct 13, 2026

Uh oh!

ask-bonk Bot commented Oct 13, 2026

Uh oh!

ask-bonk Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ask-bonk Bot commented Oct 13, 2026

Uh oh!

ask-bonk Bot commented Oct 13, 2026

Uh oh!

ask-bonk Bot commented Oct 13, 2026

Uh oh!

Uh oh!

ask-bonk Bot commented Nov 4, 2026

Uh oh!

Uh oh!

ask-bonk Bot commented Nov 4, 2026

Uh oh!

ask-bonk Bot commented Nov 4, 2026

Uh oh!

petebacondarwin commented Nov 4, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

penalosa commented Oct 13, 2026 •

edited by petebacondarwin

Loading

pkg-pr-new Bot commented Oct 13, 2026 •

edited

Loading

github-actions Bot commented Oct 13, 2026 •

edited

Loading

`cf --help` startup