Skip to content

Latest commit

 

History

History
303 lines (252 loc) · 22.1 KB

testing.md

File metadata and controls

303 lines (252 loc) · 22.1 KB

Testing

Scope

How a CLI is tested: the sandbox, no network and no real waits in tests, what kind of test to write, coverage floors, offline contracts, the test matrix, parity between CLIs, and the live checks that are not tests.

Rule ids on this page are WC-TEST-<n>. How to read a rule, how ids work and which rule wins are in the index.

Related pages: tests and the shared store are in Store; secrets in logs are in Security; typechecking the tests and the Bun smoke test are in Toolchain; which checks run on a pull request and which run after merge is in CI.

Words used here

  • Sandbox: a vitest setupFiles entry that runs before every test file and moves every path a test could write into a temporary directory.
  • Offline contract: a check that runs the built CLI from outside, with no account and no network, and compares its help, its commands --json and its MCP tool list with reviewed snapshots. The runner is cli-contract from @wirecat/cli-testing.
  • Test matrix: the page that lists every command and option of a CLI, with the test that drove it or the reason why no offline test can.
  • Full validation: the CI run on each merge to main, on a release and in the monthly run. It is not the pull request run.

Rules

WC-TEST-1 · Point every path a test could write at a temporary directory

  • Why: one test that falls back to a default path writes into the owner's config, state, cache or store.
  • Enforced by: src/testing/sandbox.test.ts · src/paths.test.ts.
  • Example: a CLI adopts the shared setup with one line in vitest.config.ts: setupFiles: ["@wirecat/cli-base/testing/setup"]. It holds for every test file, including the next one. Each file gets a fresh directory: TMPDIR, TMP, TEMP, HOME, USERPROFILE, the XDG and Windows folders, and every <APP>_CONFIG_DIR, _STATE_DIR and _CACHE_DIR already set, with the store variables (WC-STORE-14), point into it. One that is not set falls back to the sandboxed home. npm's and pnpm's update check is off (npm_config_update_notifier=false, NO_UPDATE_NOTIFIER=1): with a fresh home, a child npm would reach the registry past the network guard and print its notice into output a test parses. On macOS the directory is made under /tmp, so a Unix socket path under the sandboxed home stays within the 104-byte limit. When the file is done, after its own hooks have closed their resources, the setup restores the variables and removes the directory; when the file fails while loading or its own afterAll throws, vitest skips that step and the directory is removed as the worker ends. A CLI's own setup file, listed after the shared one, adds only what is its own, such as a default profile. Behind the sandbox, resolvePaths and the skill commands throw under vitest while HOME is the account's own, so a run that skips setupFiles (vitest run --root outside the repository) fails instead of writing into the owner's folders.
  • Exceptions: keep coverage reports and debug traces outside the sandbox, with bounded retention. Never clean another session's scratch directory as test teardown.
  • Applies to: all CLIs and the shared packages. cli-base, cli-core, every CLI and cli-template load the shared setup first in setupFiles; cli-testing does not use it yet. Since: cli-base 0.41.0. Rulings: none.

WC-TEST-2 · No test opens a real credential store

  • Why: the owner's tokens live in the real keychain, and a messenger's device keys in a file of their own; one test that forgets an option would read or write the real one.
  • Enforced by: src/keyring.test.ts · src/testing/sandbox.test.ts · whatsapp-cli: src/testing/sandbox.test.ts.
  • Example: every test passes memoryKeyring() from @wirecat/cli-base/testing. The shared setup (@wirecat/cli-base/testing/setup, WC-TEST-1) also replaces @napi-rs/keyring with a module that throws, so a forgotten option fails the test instead of reaching the keychain, and src/keyring.test.ts asserts that systemKeyring throws under it. Without the shared setup, systemKeyring itself throws under vitest. A credential store that is a file, such as WhatsApp's linked-device keys in private.db, resolves under the sandbox's state directory, and a sandbox test asserts that its path is inside the temporary root, as wa's does.
  • Exceptions: none. Until a CLI adopts the shared setup, its sandbox does not replace @napi-rs/keyring; tg and max only scope the keyring entry through environment variables.
  • Applies to: all CLIs and the shared packages. Since: cli-base 0.41.0. Rulings: none.

WC-TEST-3 · No test contacts a provider or the network

  • Why: a test that reaches the network is slow, flaky, and can act on a real account.
  • Enforced by: src/testing/network.test.ts · max-cli: src/testing/unscripted.ts.
  • Example: the shared setup (WC-TEST-1) refuses every connection to anything but localhost, any 127.x.x.x address and ::1 before a name is looked up: net and tls connects, http(s) requests, fetch and WebSocket throw an error that names the host. A test talks to a local server or a scripted adapter instead. max-cli's src/testing/mock-max.ts scripts the service: an answer per request, a refusal, or silence to test a timeout, and a request with no scripted answer fails the test that sent it. tg's scripted({...}) adapter does the same job.
  • Exceptions: a test whose subject is the network itself calls onTestFinished(allowNetwork()), with allowNetwork from @wirecat/cli-base/testing; the guard is back for the next test. A repository that does not load the shared setup has no global guard.
  • Applies to: all CLIs and the shared packages. Since: cli-base 0.41.0. Rulings: none.

WC-TEST-4 · Give a command that must not connect an adapter that throws

  • Why: a command that swallows an adapter error would otherwise pass while it still connects.
  • Enforced by: review only.
  • Example: an --offline test passes an adapter that throws "--offline must never open Telegram". When the adapter is shared across a file, it is a vi.fn, and afterAll checks expect(refuse).not.toHaveBeenCalled(). The test leaves the provider credentials out of env to show that none are needed.
  • Exceptions: none.
  • Applies to: all CLIs. Since: cli-base 0.41.0. Rulings: none.

WC-TEST-5 · No test waits for real time

  • Why: a test that sleeps a real second crosses the time limit on a slow CI runner and fails at random.
  • Enforced by: review only.
  • Example: waits go through an injected SleepLike, and fakeClock() from @wirecat/cli-base/testing records them without waiting. A test harness passes sleep: async () => {}. pnpm test:slow lists the 20 slowest tests and the 10 slowest files, and warns about each test over 3 s (WC-TEST-20). A test may wait for a local process or server it started, for example polling its status with vi.waitFor and a timeout shorter than the test's own; that waits for an event, not for time. A fixed sleep or setTimeout is still a wait for real time.
  • Exceptions: none. A new wait in the code gets a seam, never a longer test timeout.
  • Applies to: all CLIs and the shared packages. Since: cli-base 0.41.0. Rulings: dev-process:4.

WC-TEST-6 · Test mostly through the command line, with run()

  • Why: a test that drives run() checks what a person or an agent types, from parsing to output.
  • Enforced by: review only.
  • Example: most tests call run(["chats", "list", "--json"], environment) with fake effects and check stdout, stderr and the exit code. Unit tests cover pure logic with many cases. Adapter contract cases (cli-core ./testing) cover implementations that must behave the same.
  • Exceptions: none.
  • Applies to: all CLIs. Since: cli-base 0.41.0. Rulings: none.

WC-TEST-7 · Write the test that pins a promise someone could break

  • Why: a test that restates the code passes when the code is wrong; a test of a promise fails.
  • Enforced by: review only.
  • Example: reading history marks nothing read; a retried send reuses its id; a send with no answer is outcome_unknown, never failed and never sent; an ambiguous chat name is refused, not guessed.
  • Exceptions: none.
  • Applies to: all CLIs. Since: cli-base 0.41.0. Rulings: none.

WC-TEST-8 · Prove that content does not leak, with a marker and visibly fake data

  • Why: "the output has no message text" only means something when the text was there to leak; a convention alone proves nothing.
  • Enforced by: review only.
  • Example: test data is visibly fake: synthetic text ${id}, the phone number +71234567890, the device id 00000000-0000-0000-0000-000000000001, your-token-here. Content that must stay hidden carries a unique marker, and the test asserts that the marker is absent from output, errors and run events (a run's argv keeps what was typed, so a marker typed on the command line is looked for outside it), while the id is present: expect(err).not.toContain(token). The positive check stops the test from passing by recording nothing.
  • Exceptions: none.
  • Applies to: all CLIs. Since: cli-base 0.41.0. Rulings: none.

WC-TEST-9 · Leave no test's config or store change for the next test to see

  • Why: the sandbox is per file, so tests in one file share config, the send journal and the store.
  • Enforced by: review only.
  • Example: tests that only read may share one profile name, such as a file-level constant. A test that writes config or the store either uses a profile name no other test in the file uses, or undoes its write in a finally block, so a failed assertion does not skip the undo. A test that needs an empty store makes a fresh directory and passes MESSAGING_STORE.
  • Exceptions: none.
  • Applies to: all CLIs. Since: cli-base 0.41.0. Rulings: none.

WC-TEST-10 · A skip is not a pass: report the suite that ran and its counts

  • Why: a suite that ran nothing, or skipped what mattered, still exits 0.
  • Enforced by: review only.
  • Example: when you report work, name the suite that ran and give its counts: passed, failed and skipped. A skipped test is counted as skipped, never as passed. That the tests are typechecked at all is WC-TOOL-5.
  • Exceptions: none.
  • Applies to: all CLIs and the shared packages. Since: cli-base 0.41.0. Rulings: none.

WC-TEST-11 · Keep a coverage floor, and raise it; never lower it

  • Why: a floor stops coverage from falling one pull request at a time.
  • Enforced by: pnpm test:coverage · node-ci.yml:check · pr.yml:validation.
  • Example: vitest.config.ts holds thresholds for lines, statements, functions and branches over src/, a little under what the suite reaches, and perFile: { lines: 50 }. A change that falls below gets a test, not a lower number. Each excluded file has its reason written beside it.
  • Exceptions: generated code, types-only files, the entry point, and code that only runs under Bun or as a separate process, each with its reason.
  • Applies to: all CLIs and the shared packages. memo has no thresholds yet; drive and zoom have no per-file floor. Since: cli-base 0.41.0. Rulings: dev-process:4.

WC-TEST-12 · Every command and option has a test, or a written reason

  • Why: a flag that no test passes can break without anyone noticing.
  • Enforced by: node-ci.yml:check · <app>: pnpm test:matrix:check.
  • Example: under vitest, run() appends each parsed command path and option name (never a value) to coverage/argv.jsonl. pnpm test:matrix turns that into docs/dev/test-matrix.md and fails on a command or option with neither a test nor a reason. A test counts only when it runs the command through run(): a test that calls a function directly records no command line, so the matrix does not see it. A reason says what covers the command instead: the live scenario that runs it, or, for a generated group, the two tests below.
  • Exceptions: commands generated from a service's API description, such as drive's and gcal's api <method> from a Google Discovery snapshot, share one reason for the whole group (api * and its options). It names two tests: one that checks every generated command's options against the snapshot, and one that drives a generated command through run() into the executor every generated command shares. A hand-written command in the same CLI still needs its own test or live scenario.
  • Applies to: all CLIs. memo's check fails only on a stale page, not yet on an option with no test. Since: cli-base 0.41.0. Rulings: NEED-359.

WC-TEST-13 · Keep offline contracts for help, discovery, errors and the MCP tool list

  • Why: agents depend on commands --json and the MCP tool list; a change to them must be seen and reviewed.
  • Enforced by: node-ci.yml:check · <app>: pnpm contracts:check · whatsapp-cli: pnpm contracts:check.
  • Example: pnpm contracts:check runs cli-contract contracts/plan.json against the built CLI. It checks the help snapshot, that discovery is one JSON value on stdout with empty stderr, that a bad option gives the documented exit code and one JSON error on stderr, and the MCP tool names, schemas and annotations. A change is accepted with pnpm exec cli-contract contracts/plan.json --update and a reviewed diff.
  • Exceptions: none. The contracts run in full validation, on every pull request and on main (WC-CI-5, WC-CI-7).
  • Applies to: all CLIs. Since: cli-base 0.41.0. Rulings: none.

WC-TEST-14 · Check the upgrade from the previous published version

  • Why: a release that cannot read what the previous one wrote loses the owner's data.
  • Enforced by: node-ci.yml:check · <app>: pnpm contracts:upgrade.
  • Example: pnpm contracts:upgrade runs the previous published package and this build on the same synthetic state and checks that the data survives. Each store migration also has its own upgrade test (WC-STORE-9).
  • Exceptions: a CLI with no local state, or with no published version yet. Local state is data the CLI writes and reads back in a later run, such as a store, a session or a cache. The shared store counts, also when cli-core's code writes it, because the CLI's release decides which cli-core and so which migrations run: gcal's calendars and events in the store are its local state. A config file, run records, the update check, MCP logins and lock files that only cli-core writes do not count. Tests still keep them in the sandboxed home, like every other path (WC-TEST-1).
  • Applies to: all CLIs with local state. Since: cli-base 0.41.0. Rulings: none.

WC-TEST-15 · A refactor keeps commands --json and the MCP tool list as they were

  • Why: a change that is meant to move code must not change what agents see.
  • Enforced by: review only.
  • Example: capture commands --json and the MCP tool list from main before the first edit, and compare them after the last. The offline contract snapshots (WC-TEST-13) do the same job in full validation.
  • Exceptions: a refactor that changes behaviour on purpose says so in its pull request.
  • Applies to: all CLIs and cli-core. Since: cli-base 0.41.0. Rulings: none.

WC-TEST-16 · A messenger CLI matches its column of the parity manifest

  • Why: tg, max and wa promise the same commands and options where their messengers allow it.
  • Enforced by: <app>: pnpm parity:check · cli-core: parity.yml · cli-core: src/parity/manifest.test.ts.
  • Example: pnpm parity:check compares the CLI's commands --json with its column of parity.json from the cli-core version it installed. cli-core's parity.yml builds tg, max and wa from main and runs the same check in full validation.
  • Exceptions: a row marked planned passes whether the command exists or not.
  • Applies to: messenger CLIs (tg, max, wa). Since: cli-base 0.41.0. Rulings: none.

WC-TEST-17 · Merged into WC-TOOL-6

WC-TEST-18 · Check the real service by hand, never in the test suite

  • Why: a check against a real account needs a live login and can send messages to people.
  • Enforced by: <app>: bin/release · src/dev/conformance.ts, for the record. Running the scenarios stays a person's job.
  • Example: live scenarios run by hand, by the owner or with the owner's yes, on test profiles and test chats only. The cast lives in an ignored file, no profile is assumed, and the record keeps the shape of each answer, never its content. When a scenario is run again, the sentence it supports is updated. A passed run is recorded with cli-dev live record --script <its script>, and bin/release refuses a version without one (WC-REL-17).
  • Exceptions: a messenger with no test accounts, such as WhatsApp, where a second account or a second linked device would change the owner's real one, runs live scenarios on the owner's profile and only in the owner's own chat (the chat with themselves). It never writes to another person, and still runs only with the owner's yes.
  • Applies to: all CLIs that talk to a real account. Since: cli-base 0.41.0. Rulings: none.

WC-TEST-19 · A fake refuses what the real service refuses, and every provider name has a source

A fake provider, a scripted adapter or a local fake server (WC-TEST-3), checks each request the way the service does. It refuses an unknown method, an unknown field in a select, a filter or a body, an unknown parameter and a missing required field, with the error the service gives. Its vocabulary comes from the provider's published schema or a captured answer, kept as a fixture of names only, never typed from memory. The same holds for every provider field name, method name and error code in the CLI's code: each has a source, a link to the provider's documentation page or a captured answer, beside it or in the table it comes from.

  • Why: a fake that accepts anything passes a test for a request the service refuses. bitrix24-cli asked for three invented field names; every test passed, and the owner's portal refused each whole call over one unknown name.
  • Enforced by: review only, check planned: a strict fake helper in cli-base testing, built from a captured or documented schema, that refuses unknown fields and parameters.
  • Example: gcal checks every request against its generated Google Discovery manifest before it leaves (validateRequest in src/api-contract.ts), so a test cannot send a parameter Google does not know. A CRM fake refuses an unknown select name with the portal's own error, from a captured crm.item.fields answer.
  • Exceptions: a fake for an unofficial protocol with no schema keeps the names seen in captured frames as its vocabulary.
  • Applies to: all CLIs that talk to a provider. Since: cli-base 0.80.1. Rulings: none.

WC-TEST-20 · One test time limit of 15 s, and each test runs within 3 s

Every repository sets testTimeout: 15_000 in vitest.config.ts. A test that builds big data, such as a thousand meetings or a long conversation, sizes it so the test runs within 3 s on a developer machine: a fifth of the limit, because a slow CI runner takes up to five times as long. pnpm test:slow warns about each test over 3 s, so it is shrunk before a release runner fails it. cli-base's node-ci.yml shows the same warning on every run, from the merged report when the tests run in shards, and never fails the job.

  • Why: twice a release failed on a slow GitHub runner with no real wait involved, only CPU work: cli-core's test of 1200 meetings, and zoom-cli's src/answer-entities.test.ts at 6.9 s against vitest's default 5 s. Each repository had its own limit (15 s in cli-core and tg-cli, 5 s in zoom-cli, gcal-cli and max-cli), so the same test passed in one and failed in another.
  • Enforced by: pnpm test:slow · node-ci.yml:check · node-ci.yml:coverage.
  • Example: vitest.config.ts has test: { testTimeout: 15_000 }. A test that checks paging over many meetings builds as many as it needs to cross two pages, not 1200. pnpm test:slow ends with warning: 1 test takes over 3 s, … and the test's line on stderr; it still exits 0.
  • Exceptions: none. A longer timeout on one test is not the fix: cli-core #2980 gave the 1200-meeting test 60 s, and the next slower runner fails it again. The macOS and Windows jobs of cli-base's node-ci.yml pass --testTimeout=20000 because those runners are slower for every test, not for one.
  • Applies to: all CLIs and the shared packages. Since: cli-base 0.107.0. Rulings: owner, 2026-11-16.

Sources

These documents explain the rules on this page in more detail. Where one of them disagrees with this page, this page holds.