Release 1.4.0 — Sprint 18 auth, audit fixes (B1–B6), 9 new skills - #1
Merged
Conversation
From the first full audit. Two ways a test could silently pass a broken check, plus a stale model default: - assert_url with neither `contains` nor `pattern` returned "PASS — URL matches" while verifying nothing (`true && true`). It now fails, and the schema advertises an anyOf on the two matchers. (audit B5) - JUnit esc() only escaped the five XML entities, so a control char or ANSI escape in a page title / console error made the whole junit-*.xml malformed and CI importers dropped every result for the run. It now strips XML-1.0-illegal control chars and ANSI colour sequences. (audit B6) - Default AI model bumped claude-opus-4-7 -> claude-opus-4-8, and the "~90% cost reduction" prompt-cache comment corrected (the prompts are below the 4096-token cacheable minimum, so caching never fires today). Adds regression tests for each: they were untested paths, which is why the bugs shipped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cross-referencing the 51 registered MCP tools against the existing 8 skills' "Tools Used" sections surfaced a large set of shipped-but-unreachable tools — capability with no natural-language entry point. The worst gap: the marketed self-healing differentiator had no skill driving it. New skills, each unlocking the orphaned tool(s) named: - debugging-web-selectors — describe_selector + self-healing triage - verifying-web-flows — the five assert_* verbs - testing-web-error-states — route/unroute + set_network_throttle (offline) - auditing-web-security — security_audit - auditing-web-seo — seo_audit + check_broken_links - monitoring-web-regressions — get_run_history + check_regressions - authoring-web-scenarios — generate_scenario(_from_prompt) - auditing-web-pwa — pwa_audit - auditing-web-lighthouse — lighthouse_audit Each matches the house style, cites only real tool names, and states honest scope caveats. Wired into the skills README table and the testing-web-app master routing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ctor Get past the login wall (the #1 adoption blocker: every tool and scenario previously dead-ended at the login screen), plus the runner/suite correctness fixes that share these files. Authentication (storageState): - BrowserManager.launch({ storageStatePath }) -> newContext({ storageState }); saveStorageState(path). New webmobai_save_storage_state tool + storage_state_path arg on webmobai_launch_browser. Scenario `storageState` field + `saveStorageState` step, SuiteDefaults.storageState, and --storage-state / --save-storage-state on webmobai-scenario and webmobai-suite. Session files are treated as secrets — never echoed into responses/logs, and gitignored. pauseForManual scenario step: in headed mode waits (bounded) for a human to complete MFA/CAPTCHA/SSO, then continues; a fast no-op in headless. webmobai-doctor (7th binary): preflight for Node version, Playwright browser install (with the exact `npx playwright install` command), the optional Lighthouse dep, the AI key, and a storageState file's validity/expiry. Also lands (shared files with the above): - runner treats any non-success tool result as a step failure — a failed route install or a visualSnapshot capture error no longer passes as green. (audit B3) - webmobai-suite exits non-zero when a tag filter matches zero scenarios, with --allow-empty to opt out. (audit B4) Deliberately NOT built: persistent-profile / --user-data-dir / CDP-attach auth (different lifecycle, cross-browser tax, credential footgun) — the storageState slice is ~90% of the unlock at ~10% of the risk. Tests: real HTTP-origin storageState round-trip, save-tool redaction, headless pauseForManual no-op, and subprocess tests for the suite exit codes and doctor. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Counts across README/FEATURES/USER_MANUAL/mcp-server README were drifted (claimed 43/49 tools, "four/five binaries"); corrected to the real 51 tools / 7 binaries / 214 tests, and added webmobai-monitor + webmobai-doctor to the binary tables (both were missing). - New "Test behind a login" walkthrough in USER_MANUAL: the save/replay flow, the CLI flags, the saveStorageState/pauseForManual steps, and the treat-auth.json-as-a-credential warning. - README capability rows for authenticated sessions, the manual-step pause, and webmobai-doctor. - CHANGELOG [Unreleased] entry covering the Sprint 18 features and the B3–B6 correctness fixes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…er download The two remaining ship-blockers from the first full audit. B1 — desktop app was non-functional in a packaged build. The Tauri shell capability granted bare shell:allow-execute/allow-spawn with no command scope, so every Command.create (Test runner, monitor history via cat, reveal via open -R) was rejected, and plugins.shell.open (URL-only) rejected local report paths. Fixed with a scoped allow-list for exactly node / cat / open, each with validated args (NOT args:true — that would be an XSS→RCE surface under csp:null): - node-version (node --version) and node-run (node <auto-test.js> <url> <json>) as separate scope entries, since the two call sites pass different arg counts; node-run pins argv0 to a path ending in auto-test.js so -e/--eval can't be injected (later args are script args, not node flags). - cat-history (cat <*history.json>) and open-reveal (open -R <path>). - shell.open widened to file:// so Open HTML/PDF/screenshot work; openExternal wraps local paths as file:// URLs. Frontend call sites renamed to the scoped command names. Validated end to end: `cargo check` recompiles the app crate, so generate_context! parses and accepts the capability file + tauri.conf. (A packaged GUI smoke test is still advisable before shipping the .dmg.) B2 — a clean `npm install -g webmobai-mcp` ships no browsers, so every entry point except webmobai-test crashed on first launch with "Executable doesn't exist". Added a shared ensureBrowserInstalled(engine) that downloads the requested engine on first use, called from BrowserManager.launch() so the MCP server and all six CLIs are covered through one choke point (no-op once installed). auto-test's private installer now delegates to it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bump version across all manifests (root/mcp-server package.json + locks, Cargo.toml + lock, tauri.conf.json, the MCP server version string) and promote the CHANGELOG [Unreleased] section to [1.4.0]. Sprint 18 (authenticated sessions, pauseForManual, webmobai-doctor) plus the B1–B6 audit fixes; no breaking changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
celikgo
changed the title
Follow-ups from diagnosing the npm-publish failures (the primary cause — an npm token that can't bypass 2FA, giving E403 — is an account-side fix on the NPM_TOKEN secret, not a repo change). - Add timeout-minutes to both release jobs (15 for publish-npm, 60 for the desktop build) so a genuine hang fails fast instead of running to a limit. - Move the Intel desktop build off the macos-13 runner, which is unavailable to this account — the job queued for the 24h maximum and timed out every release (only the arm64 .dmg ever shipped). It now cross-compiles x86_64-apple-darwin on the Apple-Silicon runner; safe because every shipped mcp-server production dependency is arch-independent and the one native module (fsevents, via Playwright) is a universal binary. Cache is keyed by target so the two same-runner legs don't collide. - Prefix mcp-server repository.url with git+ to silence the npm publish warning. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ease.yml - Bump the actions GitHub flagged as running on the deprecated Node 20 runtime to their current Node-24 majors: actions/checkout @v4→@v7, actions/setup-node @v4→@v7, actions/upload-artifact @v4→@v7. (rust-cache is already on its latest major @v2; rust-toolchain@stable and tauri-action@v0 are moving refs and weren't flagged.) - Move CI off Node 20 (EOL April 2026) to Node 22 LTS. - Mirror the release-workflow hardening into ci.yml, which had the same landmines: the build-tauri Intel leg used the unavailable macos-13 runner (24h queue-and-timeout on every push/PR, including this one) — now cross-compiled on macos-latest like release.yml; added a per-target rust cache key and timeout-minutes to every job. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cuts v1.4.0. First-audit follow-through plus Sprint 18: real new capability (auth, manual-pause step, preflight CLI), 9 natural-language skills over previously-unreachable tools, all six audit release-blockers (B1–B6) fixed, and the version bumped across every manifest. No breaking changes.
Commits (8, logically grouped)
df2089eassert_urlvacuous pass, JUnit control-char corruption; AI model4-7→4-8936da9e63ca383pauseForManual,webmobai-doctor(+ B3/B4, shared files)03656bf4ecbd537dd2af92c2f53460c0942ci.ymlto matchrelease.ymlHeadline features
Authenticated sessions (the login-wall unlock).
storage_state_pathon launch +webmobai_save_storage_statetool; scenariostorageStatefield +saveStorageStatestep;SuiteDefaults.storageState;--storage-state/--save-storage-stateCLI flags. Session files never echoed into responses/logs, gitignored by default.pauseForManualstep — headed-mode wait for MFA/CAPTCHA/SSO, fast no-op headless.webmobai-doctor(7th binary) — preflight for Node, Playwright browser install, optional deps, API key, storageState validity.All six audit release-blockers fixed
node/cat/openwith validated args (notargs:true);shell.openwidened tofile://. Validated viacargo check(thegenerate_context!macro parses and accepts the capability file).npm i -gshipped no browsers, so 4 of 6 CLIs + the MCP server crashed on first launch.BrowserManager.launch()now downloads the requested engine on first use (one choke point, no-op once installed).routeinstall /visualSnapshotcapture error no longer passes green).webmobai-suiteexits non-zero when a tag filter matches zero scenarios (--allow-emptyto opt out).assert_urlwith no matcher is rejected, not vacuously passed.9 new skills
debugging-web-selectors(the marketed self-healing differentiator had no skill),verifying-web-flows,testing-web-error-states,auditing-web-security,auditing-web-seo,monitoring-web-regressions,authoring-web-scenarios,auditing-web-pwa,auditing-web-lighthouse.Verification
tsc+ build (all 7 binaries emit todist)tsc -b+vite buildcargo checkaccepts the capability + config at v1.4.0Before merging / releasing
webmobai-mcp404s).release.yml/ci.ymlwere hardened in2c2f534/60c0942, but confirmNPM_TOKENis set before thev1.4.0tag triggersrelease.yml, or the package stays unpublished again.🤖 Generated with Claude Code