[COVERAGE] Add manual-coverage and automation-coverage skills by DavertMik · Pull Request #96 · testomatio/skills

DavertMik · 2026-05-10T15:23:23Z

Summary

Port the standalone @testomatio/coverage-agent CLI into two agent-agnostic skills that produce the coverage.yml mapping consumed by @testomatio/reporter --filter "coverage:..." so users can run only the tests affected by a code change.

manual-coverage (test-management plugin) — maps manual *.test.md cases to source files; outputs coverage.manual.yml.
automation-coverage (test-automation plugin) — maps automated e2e tests (Playwright, Cypress, WebdriverIO, CodeceptJS, Puppeteer, Appium) to source files; outputs coverage.e2e.yml.

Both skills delegate discovery to the new project-scan skill (frameworks, test inventory, project overview) and only own the source-file → ID mapping logic. manual-coverage delegates pulling cases to sync-cases.

Also:

Wires both skills into the matching plugin marketplaces via symlinks.
Adds a Coverage Mapping Flow section to testomatio-flow and lists both skills in the orchestrator's specialized-skills table.
Symlinks project-scan into the test-automation plugin so automation-coverage can rely on it.
Updates the README skills table.

Test plan

Lint: manual-coverage/SKILL.md and automation-coverage/SKILL.md parse with valid frontmatter (name, description).
Symlinks resolve: ls -la plugins/test-management/skills/manual-coverage, ls -la plugins/test-automation/skills/automation-coverage, ls -la plugins/test-automation/skills/project-scan.
End-to-end manual: in a sample repo with manual-tests/*.test.md containing @S… / @T… IDs, prompt the agent "create coverage mapping for our manual tests" → expect coverage.manual.yml written with realistic suite/test/tag mappings, no other files modified.
End-to-end automation: in a Playwright project with IDs already in test names, prompt "map e2e tests to source files" → expect coverage.e2e.yml.
Reporter integration: npx @testomatio/reporter run "npx playwright test" --filter "coverage:file=coverage.e2e.yml,diff=main" selects only mapped tests against a known small diff.
Flow: invoking testomatio-flow after sync-cases pull surfaces manual-coverage as a next-step suggestion; after reporter-setup it surfaces automation-coverage.

🤖 Generated with Claude Code

Port the standalone @testomatio/coverage-agent CLI into two agent-agnostic skills that produce the coverage.yml file consumed by `@testomatio/reporter --filter "coverage:..."` to run only the tests affected by a code change. - manual-coverage (test-management): map manual *.test.md cases to source files, output coverage.manual.yml. - automation-coverage (test-automation): map e2e tests (Playwright, Cypress, WebdriverIO, CodeceptJS, Puppeteer, Appium) to source files, output coverage.e2e.yml. Both skills delegate discovery to the new project-scan skill (frameworks, test inventory, project overview) and only own the source-file -> ID mapping logic. manual-coverage delegates pulling cases to sync-cases. Wires the new skills into both plugin marketplaces, adds a Coverage Mapping Flow section to testomatio-flow, and links project-scan into the test-automation plugin so automation-coverage can rely on it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

… reference The Testomat.io classical tests Markdown format is already documented in skills/generate-cases/references/test-case-format.md (canonical, more thorough). Both manual-coverage and generate-cases live in the test-management plugin, so a relative cross-skill link works for any user installing the plugin. Note: COVERAGE_FILE_FORMAT.md is intentionally still duplicated in both manual-coverage and automation-coverage because those two skills live in *different* plugins (test-management vs test-automation), and a cross-plugin reference would break for users who install only one. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Both coverage skills now coexist in both plugins (test-management and test-automation), mirroring how project-scan is wired. With both skills present in either install context, the shared coverage YAML reference no longer needs to be duplicated. - skills/manual-coverage/references/COVERAGE_FILE_FORMAT.md is now a symlink to skills/automation-coverage/references/COVERAGE_FILE_FORMAT.md (canonical — has the full grammar plus GitHub Actions example). - plugins/test-management/skills/automation-coverage and plugins/test-automation/skills/manual-coverage added so the symlink resolves regardless of which plugin a user installs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

- Drop manual-coverage from test-automation plugin: manual coverage is a test-management concern, not an automation one. - Keep automation-coverage in BOTH test-management and test-automation plugins: both coverage skills require Testomat.io-formatted cases in the system, which is a test-management precondition; the skill is also legitimately useful in the test-automation workflow. - Reflect the dual home in README: automation-coverage now appears under both Test Management and Automation tables. The cross-skill symlink at skills/manual-coverage/references/COVERAGE_FILE_FORMAT.md still resolves: both skills exist together in test-management plugin. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

mykhailiukVitalii

Found some quick updates to be more clear in this skill's description
@DavertMik FYI

mykhailiukVitalii · 2026-05-11T17:55:38Z

+| Framework    | Filename patterns                                         | Imports                              | Test syntax                              |
+| ------------ | --------------------------------------------------------- | ------------------------------------ | ---------------------------------------- |
+| Playwright   | `*.spec.ts`, `*.spec.js`, `*.test.ts`                     | `@playwright/test`                   | `test(...)`, `test.describe(...)`        |
+| Cypress      | `*.cy.js`, `*.cy.ts`                                      | `cypress`                            | `describe(...)`, `it(...)`, `context(...)` |
+| WebdriverIO  | `*.test.js`, `*.e2e.js`                                   | `@wdio/cli`, `webdriverio`           | `describe(...)`, `it(...)`               |
+| Puppeteer    | `*.test.js`                                               | `puppeteer`                          | `describe(...)`, `it(...)`               |
+| CodeceptJS   | `*_test.js`, `*.test.js`                                  | `codeceptjs`                         | `Feature(...)`, `Scenario(...)`          |
+| Appium       | `*.spec.js`, `*.e2e.js`                                   | `appium`, `webdriverio`              | `describe(...)`, `it(...)`               |
+| Mocha (e2e)  | `*.test.js`, `*.spec.js`                                  | `mocha`                              | `describe(...)`, `it(...)`               |
+| Jest (e2e)   | `*.test.js`, `*.spec.js`                                  | `jest`                               | `describe(...)`, `it(...)`, `test(...)`  |


Doesn't it seem too complicated?

mykhailiukVitalii · 2026-05-11T18:20:21Z

+| `sync-cases`           | Synchronize Markdown test scenarios between local project and Testomat.io                            |
+| `manual-coverage`      | Map manual test cases to source files; generate `coverage.manual.yml` for affected-only runs         |
+| `automation-coverage`  | Map e2e tests to source files; generate `coverage.e2e.yml` to run only the tests affected by a diff  |
+| `testomatio-flow`      | Orchestrate complete test case lifecycle: generate, improve, analyze coverage, upload to TMS         |


Suggested change

| `testomatio-flow` | Orchestrate complete test case lifecycle: generate, improve, analyze coverage, upload to TMS |

| `testomatio-flow` | Orchestrate complete test case lifecycle: generate, improve, analyze coverage, upload to TMS |

| `project-scan` | Scan project source code to inventory languages, frameworks, and existing tests |

Co-authored-by: Vitalii Mykhailiuk <82405549+mykhailiukVitalii@users.noreply.github.com>

DavertMik · 2026-05-11T18:40:53Z

+## Related skills
+
+- `reporter-setup` — install `@testomatio/reporter` and import tests via `check-tests`.
+- `sync-cases` — pull/push manual cases (the manual-coverage counterpart).


mentijon sync cases and its ref there

DavertMik · 2026-05-11T18:41:02Z

+2. 🗺️ Map e2e tests to source files for affected-tests-only CI (with `automation-coverage` skill)
 ```

+### **Coverage Mapping Flow (run only affected tests)**


remove this section

DavertMik · 2026-05-11T18:41:29Z

@@ -0,0 +1 @@
+../../automation-coverage/references/COVERAGE_FILE_FORMAT.md


overage file format MUST BE inside test managemtn

- Revert all changes to the test-automation plugin: remove the automation-coverage and project-scan symlinks I had added there. Both coverage skills (and project-scan) live in test-management only — both require Testomat.io-formatted cases stored in the system. - README: automation-coverage listed only under Test Management. - Move the canonical COVERAGE_FILE_FORMAT.md into manual-coverage (a test-management skill); automation-coverage now symlinks to it. - testomatio-flow: remove the "Coverage Mapping Flow" section and the e2e-coverage follow-up bullet under the reporter-setup flow. - E2E_FRAMEWORKS.md: link the sync-cases Testomat.io CLI reference next to the sync-cases mention. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The agent was spawning a Python heredoc to pull suite/test IDs and tags out of .test.md files. These projects are mostly JS environments and a stray interpreter is an unnecessary risk. Both coverage skills now: - carry a "DO NOT write or execute ad-hoc scripts (Python, Node, shell heredocs)" line in CRITICAL CONSTRAINTS; - spell out the extraction approach: read the test files directly with the file tool, and for large sets narrow down with simple `grep` one-liners (examples included) instead of writing a parser. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The agent followed up generation by spawning a Python heredoc to parse and validate the produced coverage.*.yml. The "Save the coverage file" step is now "Save and validate", with an explicit no-scripts validation checklist: re-read the file, cross-check keys against the project-scan inventory, cross-check IDs against the set already extracted, and don't emit empty entries. `npx js-yaml <file>` is offered as the JS-native fallback if a real parser check is genuinely wanted. The CRITICAL CONSTRAINTS no-scripts line now covers validation, not just parsing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

@t

The agent kept improvising fragile parsers (Python heredocs for ID extraction, Python/Node heredocs for YAML validation). Replace the inline "recipe" approach with committed, zero-dep ESM helpers the skill just calls: - manual-coverage/scripts/parse-manual-tests.mjs — per .test.md file: suite ID + title, test IDs + titles, tags (flags files without IDs). - manual-coverage/scripts/validate-coverage.mjs — validates a coverage.*.yml: structure, file keys exist (glob: base dir exists), no empty entries; lists referenced @S/@T/tags; exits non-zero on problems. (canonical) - automation-coverage/scripts/parse-tests.mjs — per test file: @S/@t IDs and @tags found in describe/it/test/Scenario/Feature names. - automation-coverage/scripts/validate-coverage.mjs — symlink to the manual-coverage validator (same YAML grammar). SKILL.md updates: steps now say `node scripts/<name>.mjs …`; CRITICAL CONSTRAINTS reworded to "use the bundled scripts; never use Python; don't hand-roll parsers" (Node `-e` one-liner is the ceiling if a script doesn't fit); each skill gets a "Bundled scripts" table; the shared COVERAGE_FILE_FORMAT.md "Validating" section points at the script. Scripts smoke-tested against a fixture. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…e repo The coverage skills run inside the user's source repo, so pulled manual cases (and cloned e2e-test repos) must never land in a tracked folder there. They now go into the gitignored cache `.testclaw-context/` — mirroring how project-scan already pulls source code into `.testclaw-context/code/` when it runs inside a manual-tests repo. - project-scan: documents `.testclaw-context/` as the cache directory (manual-tests / code / e2e-tests subdirs), ensures it's in the project .gitignore (only if missing), scans `.testclaw-context/manual-tests/` on re-runs, treats `.testclaw-context/code/` as source despite the gitignore, and states it changes no tracked file on a source repo. - sync-cases: "where to pull — two cases": to edit -> tracked folder (manual-tests/, default, unchanged); to analyze -> gitignored `.testclaw-context/manual-tests/`. - manual-coverage: no-cases branch pulls into `.testclaw-context/manual-tests/`; pulled cases stay cached there; only writes coverage.manual.yml + one .gitignore line; parse helper can target the cache. - automation-coverage: clones an external e2e repo into `.testclaw-context/e2e-tests/`; same write discipline. Wording simplified throughout (ran the writing-clearly skill): plain, short sentences; dropped "round-trip", "idempotent", "write discipline", "hand-roll", etc. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

@s

The bundled helpers were too big — a 100-line parser script is the kind of thing the agent may balk at running. Replace them with plain suggestions and one tiny script: - Deleted parse-manual-tests.mjs, parse-tests.mjs, and validate-coverage.mjs. - Extracting IDs/tags is now a `grep` one-liner in the SKILL.md (`grep -rnE 'id:[[:space:]]*@s' <dir>`, `grep -rnoE '@[ST][0-9a-f]{8}' <dir>`, etc.) or just reading the files. - Validating the YAML is `npx js-yaml <file>` (parses?) plus the one remaining script, `scripts/check-coverage.mjs` (~30 lines, zero deps): flags keys whose path is missing on disk, flags keys with no identifiers, lists the @S/@T/tag IDs referenced. automation-coverage's copy is a symlink to it. - CRITICAL CONSTRAINTS reworded: "Don't write scripts. Never use Python." with a one-line `node -e` escape hatch. Smoke-tested check-coverage.mjs (real + symlink) and `npx js-yaml` (accepts valid YAML, rejects malformed). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Per review: stop line-parsing the YAML by hand when js-yaml is right there. `npx js-yaml file.yml` already prints JSON and exits non-zero on malformed YAML, so the whole check is now one pipe, run from the project root: npx js-yaml coverage.manual.yml | node scripts/check-coverage.mjs check-coverage.mjs is now ~25 readable lines: JSON.parse(stdin), iterate entries, flag missing-path keys and empty keys, list the @S/@T/tag IDs. Updated both SKILL.md files and COVERAGE_FILE_FORMAT.md to the pipe form. Smoke-tested via the real path and the automation-coverage symlink. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

DavertMik requested review from mykhailiukVitalii and olexandr13 May 10, 2026 15:23

DavertMik and others added 3 commits May 10, 2026 18:26

mykhailiukVitalii reviewed May 11, 2026

View reviewed changes

Apply suggestions from code review

b69c761

Co-authored-by: Vitalii Mykhailiuk <82405549+mykhailiukVitalii@users.noreply.github.com>

DavertMik commented May 11, 2026

View reviewed changes

DavertMik and others added 5 commits May 11, 2026 22:47

mykhailiukVitalii mentioned this pull request May 12, 2026

Skill: Ship coverage agent as skill #2

Open

DavertMik and others added 2 commits May 13, 2026 01:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[COVERAGE] Add manual-coverage and automation-coverage skills#96

[COVERAGE] Add manual-coverage and automation-coverage skills#96
DavertMik wants to merge 12 commits into
masterfrom
coverage-skills

DavertMik commented May 10, 2026

Uh oh!

mykhailiukVitalii left a comment

Uh oh!

mykhailiukVitalii May 11, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mykhailiukVitalii May 11, 2026

Uh oh!

Uh oh!

Uh oh!

DavertMik May 11, 2026

Uh oh!

DavertMik May 11, 2026

Uh oh!

DavertMik May 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	\| `testomatio-flow` \| Orchestrate complete test case lifecycle: generate, improve, analyze coverage, upload to TMS \|
	\| `testomatio-flow` \| Orchestrate complete test case lifecycle: generate, improve, analyze coverage, upload to TMS \|
	\| `project-scan` \| Scan project source code to inventory languages, frameworks, and existing tests \|

		@@ -0,0 +1 @@
		../../automation-coverage/references/COVERAGE_FILE_FORMAT.md No newline at end of file

Conversation

DavertMik commented May 10, 2026

Summary

Test plan

Uh oh!

mykhailiukVitalii left a comment

Choose a reason for hiding this comment

Uh oh!

mykhailiukVitalii May 11, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mykhailiukVitalii May 11, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

DavertMik May 11, 2026

Choose a reason for hiding this comment

Uh oh!

DavertMik May 11, 2026

Choose a reason for hiding this comment

Uh oh!

DavertMik May 11, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants