Skip to content

Latest commit

 

History

History
154 lines (128 loc) · 8.59 KB

CONTRACTS.md

File metadata and controls

154 lines (128 loc) · 8.59 KB

Offline CLI contracts

cli-contract drives built Node command-line tools and their MCP stdio servers from the outside. Every WireCat CLI keeps its plans and reviewed snapshots in contracts/ in its own repository: MAX, Telegram, WhatsApp, Zoom, Drive and Memo, and runs them in full validation. A CLI on cli-base node-ci.yml with cli: true runs them as the named steps contracts:check, contracts:upgrade and security:no-leak. The shared runner is also exported as runContracts from @wirecat/cli-testing.

Toolkit distribution

Repositories pin @wirecat/cli-testing from npm as a development dependency, with its SHA512 integrity recorded in each CLI lockfile. Trusted publishing was verified by release run 38092344081, which published the exact validated artifact and created tag v0.2.0. Repository installs and contract checks do not require a local npm login.

The initial rollout used the 0.2.0 toolkit preview while publisher configuration was pending. That preview remains available for provenance; its package integrity matches the npm release.

Checks

  • Help output matches a reviewed snapshot.
  • Discovery is exactly one JSON object or array on stdout, with empty stderr.
  • Invalid options and command paths return the documented exit code and one JSON error on stderr, with empty stdout and the expected error code.
  • MCP initialization and tools/list work through the SDK. Tool names, the structure of each input schema, and annotations match a reviewed snapshot. Descriptions (of tools and inside schemas), $schema, output schemas and tool ordering are not frozen: they change with each cli-core release and only made every CLI adopt it. A snapshot written with output schemas still passes; --update rewrites it in the shorter form.

The first plans cover shared metadata commands and errors. They do not claim coverage of every command, store migration, messenger protocol, HTTP MCP transport, or live-account behavior. Those remain separate testing tasks.

Isolation

Each run starts with a fresh temporary home, config/state/cache directories, and synthetic store path. Tokens, application credentials, proxy configuration, and the owner's keyring environment are not inherited. A Node preload guard rejects common network, keyring, and child-process entry points, and native addons are disabled. An attempted external operation fails the run even when the command catches its exception. The runner accepts only the current Node executable.

This is a regression-test guard for trusted CLI builds, not a sandbox for arbitrary untrusted code. Plans must use offline metadata operations. The MCP client initializes and lists tools; it calls a tool only in the deny check, for a command that profile denies. Temporary directories and server processes are closed afterward. Cases default to a five-second process limit; MCP uses one five-second request budget across initialization and pagination. Process output has an eight-megabyte limit.

Run and review

From a CLI repository with its build present:

pnpm contracts:check
pnpm exec cli-contract contracts/plan.json --update
git diff -- contracts/

Ordinary checks never update snapshots. A missing or changed snapshot fails validation. Use --update only when deliberately reviewing a contract change. Output normalization is limited to line endings and JSON object-key ordering; renamed tools and changed parameter schemas remain visible. Help snapshots retain command and option descriptions.

Contract checks run during full release, monthly, and manual validation. Local commit/push hooks and ordinary PR CI keep their existing budgets. The runner returns exit 0 for a passing report, 1 for contract failures, or 2 for setup errors. Reports contain case identifiers and failure reasons; they do not copy CLI stdout/stderr into public logs.

Plan format

{
  "command": "node",
  "args": ["../dist/bin/example.js"],
  "cases": [
    {
      "id": "discovery",
      "args": ["commands", "--json"],
      "exit": 0,
      "stream": "stdout",
      "kind": "json"
    }
  ],
  "mcp": {
    "args": ["mcp"],
    "snapshot": "mcp-tools.json"
  }
}

Relative executable arguments and snapshot paths resolve from the plan directory. Add contains for a required output fragment, errorCode for a structured error, and snapshot to freeze text such as help. MCP is optional for command-line tools without a server.

Deny check

"deny": { "profile": "personal" } checks that a deny in the profile stops each command that reaches the service, on the command line and over MCP (WC-PERM-23). It reads commands --json and takes each leaf whose effects.remote is read or write, except a persistent one such as mcp serve. Its arguments are the words after its path in its first example that has no --dry-run, or x for each required argument and option. <profile> <path> --dry-run --json names the command's key; housekeeping such as session and doctor names none and is left out. <profile> config show --json names the config file, inside the fresh root.

  • deny-keys fails when no read names a key, or no write does in a CLI that has writes: the program then dropped its permission table, as one CLI's did.
  • deny:<path> writes deny for that key alone and runs the command: exit 5, permission_error on stderr, nothing on stdout, and no network, keyring or child-process access.
  • deny-mcp denies every key and starts <profile> mcp serve. Each command not marked mcp: false or interactive is called through the server's *_read or *_write tool and must come back permission_error. A validation_error that carries the arguments' schema is retried once with the required arguments filled; a second validation_error fails, since it hides whether the deny holds.

Stateful plans

fixtures seeds {path, text} files inside the fresh profile before cases. json checks object subsets with exact array lengths; capture maps names to JSON pointers such as /items/0/id. Use {{name}} in later arguments and expectations. Whole-value JSON placeholders preserve the captured string/number type; {{root}} identifies the fresh root. Missing/repeated names fail. Per-case files check reviewed text, JSON subsets, original fixture bytes (unchanged: true) or SHA256. Paths cannot escape the root or pass through symlinks.

previous: {} tests the upgrade from the newest published release at or below the version in the package.json above the plan: cli-contract asks npm for the versions, installs that one with pnpm into node_modules/.cache/wirecat-contract-previous/ (once per version, without install scripts), and reports it as previous in its result. A release candidate whose package.json already holds the next version tests from the last release; a pull request tests from the version on npm. Name the executable with previous: {"bin": "<name>"} when the package has more than one target. This step needs the registry; the cases themselves stay offline. previous: {entry, version} still pins an installed Node CLI entry, for a plan whose seed script imports a release's own modules. A case with previous: true uses that binary. entry can name a reviewed local synthetic fixture generator. Entry paths resolve relative to the plan, while fixture and artifact paths stay relative to the temporary root. diagnostic permits one exact reviewed stderr string on a stdout case; leak checks still apply. diagnosticPattern does the same with a regular expression, for a message that carries a number or a version: it is matched against the whole normalized stderr (anchored, no flags), and a mismatch names the pattern and the stderr received. It replaces diagnostic, which it cannot be combined with, and needs stream: "stdout". An invalid expression or both fields fail the whole plan before any case runs. In JSON the backslash is doubled:

{
  "id": "store-newer-than-cli",
  "args": ["upgrade", "--check", "--json"],
  "exit": 0,
  "stream": "stdout",
  "kind": "json",
  "diagnosticPattern": "the store moved to schema \\d+; every other WireCat CLI that uses it needs a version that speaks it — run `<cli> upgrade` for each\n"
}

canaries names synthetic sensitive values and artifacts selects generated logs to scan. allowSensitiveOutput: true permits only canaries in requested stdout data; credential and phone patterns are still checked. See upgrade checks for the concrete CLI coverage.