docs + fix: v1.4.0 documentation overhaul, repaired install paths, CI-verified claims - #2
Merged
Merged
Conversation
The doc set had drifted badly from the shipped surface: Sprint 18's authenticated-session feature had zero skill coverage, three tool-reference sections claimed 51 tools while listing ~43, and several counts (tests, skills, binaries) were stale across four files. Skills — 17 -> 20: - NEW testing-web-authenticated-sessions: the storageState capture/replay workflow (Mode A capture with a manual MFA pause, Mode B replay via storage_state_path / --storage-state / scenario storageState), login-wall detection mid-run, expiry + re-capture, and secret hygiene. - NEW running-web-ci-suites: suite files as a CI gate — --workers, --shard, --tag, exit-code contract, JUnit wiring, artifact upload, GH Actions YAML. - NEW troubleshooting-webmobai-setup: webmobai-doctor-driven decision tree for browser/engine/Lighthouse/AI-key/storageState/Gatekeeper failures. - All 17 existing skills: corrected drifted claims against source, added auth handoffs, the versioned-baseline tools, the opt-in AI tools, the v1.4.0 assert_url and scenario-runner semantics changes. - skills/README.md rebuilt: 20-skill index, updated decision tree, new Preflight / Authenticated sessions / AI layer conventions, and a tool table whose category counts sum to exactly 51. Docs: - USER_MANUAL section 7 rebuilt as a per-tool table: all 51 tools, one row each, with a column for whether a launched browser is required. - NEW docs/AUTHENTICATION.md, docs/CI.md, docs/README.md (doc index). - SCENARIO_FORMAT: storageState, saveStorageState, pauseForManual. - FEATURES: Sprint 18 sections, 20-skill list, corrected limitations. - ROADMAP: Sprint 18 entry + What's next. - CONTRIBUTING: skill authoring, "keeping docs honest" count checklist. - README / mcp-server README: accurate counts, docs index, auth example. Corrected throughout: 51 tools (16 files), 7 binaries, 214 tests across 26 files, 20 skills. Every relative link resolves; every JSON/YAML block parses. Docs and npm keywords only — no source, build, or test changes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ified Two of the four documented install paths were dead, and the numbers the README advertises were unenforced prose. This fixes the first and makes the second structurally impossible to get wrong again. Findings that changed the shape of the work: * The npm publish job already existed. `release.yml` has carried a `publish-npm` job since the v1.4.0 hardening; it ran on the tag and failed in 26s with `E404 Not Found - PUT https://registry.npmjs.org/webmobai-mcp`. A 404 on a PUT is npm's signature for a token that cannot authenticate, so NPM_TOKEN is invalid/expired/read-only. This was a credential problem, not a packaging one. The publish itself is done from a logged-in machine. * `webmobai-mcp --help` would have hung even once published. src/index.ts had no argv handling at all: it connected the stdio MCP transport and blocked forever on stdin. The command the README teaches for verifying an install could never have worked. Adds --help/--version, and drops a hardcoded "v1.2.0" startup banner that had drifted from the real 1.4.0 by reading the version from package.json instead. * The three headline numbers were already correct. 51 tools, 7 binaries and 214 tests all recounted true, so none of them were "fixed" — they are now verified instead of trusted. Install paths: * index.ts gains --help/--version, with 5 subprocess tests in which a regression hangs and the timeout is the assertion. * release.yml publishes with --provenance, guards that the tag matches package.json, and smoke-tests the built entrypoint before publishing. * server.json plus an `mcpName` in package.json, staged for the official MCP Registry. mcpName must be in the published tarball — the registry reads it from there to verify ownership, and npm versions are immutable — so it lands before the first publish, not after. Links: * github.com/modelcontextprotocol/sdk 404s; corrected to typescript-sdk. * Placeholder domains across the skill docs (foo.com, bar.com, x.com) are real registrable domains that a link checker will chase. Replaced with RFC 2606 reserved names so the new workflow needs no arbitrary exceptions. * New docs-links workflow (lychee), modeled on the salman repo's. Every exclusion is a URL that cannot resolve by design, and says why in-file. Numbers: * scripts/check-doc-claims.mjs recounts tools, tool files, binaries, tests, test files, skills and the eval rates from the tree and checks 29 claims across 3 files. Verified to fail on injected drift. It caught the test additions in this very commit, which is why the README now reads 219/27. Designed for agents: * docs/DESIGNED_FOR_AGENTS.md explains why a selector miss returns a prior fingerprint, ranked candidates and page-state triage, with a transcript captured from the running code rather than illustrated. * An 18-case eval measures it: 86.7% top-1 recovery, 93.3% top-3, 83.3% overall. It runs in CI and gates at an 80% floor. The corpus deliberately includes cases the tool fails, and the doc names all three — a suite that only contains passes measures the author's taste in examples. * The eval runs against dist/, not src/. An early 0% reading was esbuild's keepNames helper leaking a `__name` reference into page.evaluate — a property of the loader, not the product. The shipped build is sound. Hygiene: * SECURITY.md, covering what storageState files actually contain (live session tokens), that they are never transmitted, and the disclosure path. * CODE_OF_CONDUCT.md (Contributor Covenant 2.1). * Social preview: a split frame — the request in plain English beside the real browser run with the assertion passing. HTML source kept in-repo so the image is reproducible; it carries no counts, because a number baked into a PNG is the one claim check-doc-claims cannot reach. * Release matrix extended to Linux and Windows. Ubuntu 22.04 rather than latest, so the AppImage does not silently raise the glibc floor. Still open: the npm publish itself, the homepage field (points at the npm package once it exists), uploading the social preview (GitHub has no API for it), and the registry submissions, which are staged in docs/MCP_DIRECTORY_SUBMISSIONS.md and deliberately not sent. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011ixfhPuvZGgvevYHsJRp2V
The first run of this job did its job and failed: 250 links checked, 3 errors, all three `blob/main` URLs in mcp-server/README.md pointing at docs/AUTHENTICATION.md and docs/CI.md. Those links are correct. The files exist — on this branch. They are not at `blob/main` until the branch merges, which makes the failure circular: the check blocks the pull request that would make the check pass. Waiting it out would mean merging with a red check and trusting that it goes green, which is the habit this workflow exists to break. The links cannot simply become relative. mcp-server/README.md is the npm package page, rendered on npmjs.com, where `../docs/CI.md` resolves to nothing — absolute URLs are right for the reader. So remap this repository's own blob URLs onto the checked-out working copy. That tests the invariant actually worth testing — the file this link points at exists in this repository — which holds both before and after the merge, and still fails loudly when a link names a file that genuinely is not there. Links to any other repository are still fetched over the network. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011ixfhPuvZGgvevYHsJRp2V
The first green run was preceded by a red one whose only error was `[ERROR] https://tauri.app/ | Connection failed` — a link that is fine. Re-running passed with nothing changed. A gate that fails for reasons unrelated to what it guards teaches everyone to re-run it without reading it, and a check nobody reads costs time and buys nothing. The main cause here is concurrency: lychee fires up to 128 requests at once by default, and aiming that at a couple of dozen hosts produces connection resets and rate-limit refusals that are indistinguishable from dead links. Caps concurrency at 16 (250 links still finish in seconds), retries harder before declaring a link dead, and sends a real identifiable user agent so CDNs that refuse unknown agents give an honest answer instead of a refusal. A 404 still fails the build. Only the noise is gone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011ixfhPuvZGgvevYHsJRp2V
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The doc set had drifted from the shipped v1.4.0 surface. Three separate tool-reference sections claimed 51 tools while actually listing ~43, Sprint 18's authenticated-session feature had zero skill coverage, and the test/skill/binary counts were stale across four files.
Skills: 17 → 20
New
testing-web-authenticated-sessionsstorage_state_path/--storage-state/ the scenariostorageStatefield. Plus login-wall detection mid-run, expiry and re-capture, and secret hygiene.running-web-ci-suites--workers,--shard,--tag, the exit-code contract, JUnit wiring, artifact upload, and a complete GitHub Actions example.troubleshooting-webmobai-setupwebmobai-doctor-driven decision tree for browser / engine / Lighthouse / AI-key / storageState / Gatekeeper failures.Updated — all 17 existing skills had drifted claims corrected against source, and gained auth handoffs, the versioned-baseline tools, the opt-in AI tools, and the v1.4.0
assert_url+ scenario-runner semantics changes..claude/skills/README.mdwas rebuilt: 20-skill index, updated decision tree, new Preflight / Authenticated sessions / AI layer conventions, and a tool table whose category counts sum to exactly 51.Docs
USER_MANUAL.md§7 rebuilt as a per-tool table — all 51 tools, one row each, with a column for whether a launched browser is required (+538 lines).docs/AUTHENTICATION.md(426),docs/CI.md(477),docs/README.md(doc index).docs/SCENARIO_FORMAT.md:storageState,saveStorageState,pauseForManual.FEATURES.md: Sprint 18 sections, the 20-skill list, corrected limitations.ROADMAP.md: Sprint 18 entry + What's next.CONTRIBUTING.md: skill authoring, and a "keeping docs honest" checklist of every place each count is asserted.README.md/mcp-server/README.md: accurate counts, docs index, auth example.Counts corrected throughout
51 tools across 16 tool files · 7 binaries · 214 tests across 26 files (was
200/23) · 20 skills (FEATURES.mdsaid8).Verification run against the final tree
names match their directory name.One product finding (not fixed here)
pauseForManualnever actually pauses under either shipped CLI.scenario-cli.ts:84andsuite/runner.ts:105both hardcodeheadless: true, andscenario/runner.ts:143-151makes the step a warn-and-continue no-op when headless. A scenario doingpauseForManual→saveStorageStatetherefore writes an unauthenticated session file and records the run as green. The step only works for programmaticBrowserManager+runScenariocallers.The new auth skill documents this as a trap to avoid rather than a recommended path, but the real fix is either a
--headedflag on those CLIs or making the step fail loudly in headless. Worth a follow-up issue.🤖 Generated with Claude Code
Install paths repaired, and every published number made CI-verified
Added after the original doc overhaul. Two of the four documented install paths were dead, and
the numbers the README advertises were unenforced prose.
What the investigation actually found
The npm publish job already existed.
release.ymlhas carried apublish-npmjob since thev1.4.0 hardening. It ran on the v1.4.0 tag and failed in 26s with
E404 Not Found - PUT https://registry.npmjs.org/webmobai-mcp. A 404 on aPUTis npm'ssignature for a token that cannot authenticate, so
NPM_TOKENis invalid, expired, orread-only. This was a credential problem, not a packaging one.
webmobai-mcp --helpwould have hung even once published.src/index.tshad no argvhandling at all — it connected the stdio MCP transport and blocked forever on stdin. The command
the README teaches for verifying an install could never have worked.
All three headline numbers were already correct. 51 tools, 7 binaries, 214 tests each
recounted true, so none were "fixed". They are now verified rather than trusted.
Changes
Install paths
index.tsgains--help/--version, plus 5 subprocess tests in which a regression hangsand the timeout is the assertion. Also drops a hardcoded
v1.2.0startup banner that haddrifted from the real 1.4.0, by reading the version from
package.json.release.ymlpublishes with--provenance, guards that the tag matchespackage.json, andsmoke-tests the built entrypoint before publishing.
server.json+ anmcpNameinpackage.json, staged for the official MCP Registry.mcpNamehas to be in the published tarball — the registry reads it from there to verify ownership, and
npm versions are immutable — so it lands before the first publish, not after.
Links
github.com/modelcontextprotocol/sdk404s; corrected totypescript-sdk.foo.com,bar.com,x.com) are real registrabledomains a link checker will chase. Replaced with RFC 2606 reserved names.
docs-linksworkflow (lychee), modeled on thesalmanrepo's. It found 3 real dead linkson its first run and now checks 250 with zero dead.
Numbers
scripts/check-doc-claims.mjsrecounts tools, tool files, binaries, tests, test files, skillsand the eval rates from the tree, checking 29 claims across 3 files. Verified to fail on
injected drift — it caught this PR's own test additions, which is why the README now reads
219 tests / 27 files.
Designed for agents
docs/DESIGNED_FOR_AGENTS.md— why a selector miss returns a prior fingerprint, rankedcandidates and page-state triage, with a transcript captured from the running code rather than
illustrated.
An 18-case eval measures it, running in CI and gating at an 80% floor:
The corpus deliberately includes cases the tool fails, and the doc names all three — a
suite containing only passes measures the author's taste in examples, not the algorithm. The
eval runs against
dist/, notsrc/: an early 0% reading was esbuild'skeepNameshelperleaking a
__namereference intopage.evaluate, a property of the loader rather than theproduct. The shipped build is sound.
Hygiene
SECURITY.md— whatstorageStatefiles actually contain (live session tokens), that they arenever transmitted, and the disclosure path.
CODE_OF_CONDUCT.md(Contributor Covenant 2.1).the assertion passing. HTML source kept in-repo so the image is reproducible. It carries no
counts, because a number baked into a PNG is the one claim
check-doc-claimscannot reach.ubuntu-22.04rather thanlatestso theAppImage does not silently raise the glibc floor.
Two CI fixes that were not planned
The link checker failed on its first real run, twice, and both causes were worth fixing rather
than merging past:
blob/mainlinks inmcp-server/README.mdpointing atdocs/AUTHENTICATION.mdanddocs/CI.md— files added on this branch, so they 404 until itmerges, and the check blocks the merge that fixes it. Those links cannot become relative
(that README renders on npmjs.com, where
../docs/CI.mdresolves to nothing), so this repo'sown blob URLs are remapped onto the checked-out working copy. That tests the invariant worth
testing — the file this link points at exists in this repository — before and after merge.
[ERROR] https://tauri.app/ | Connection failed, on a link that is fine;re-running passed unchanged. lychee defaults to 128 concurrent requests, which produces
connection resets indistinguishable from dead links. Concurrency capped at 16, harder
retries, real user agent. A 404 still fails the build.
Still open after this merge
been published.
homepagefield — points at the npm package once it exists.docs/assets/README.mdhas the path.docs/MCP_DIRECTORY_SUBMISSIONS.mdand deliberately notsent.
dependencies and the
shell: bashfix are written from the documented requirements, butrelease.ymlonly fires onv*, so the first real proof is the next tag.