ci: validate skills, manifest sync, and api spec drift

[dg-coreylweathers]

Why

.github/workflows/ contained exactly one file, context7.yml, which only refreshes Context7 on release publish. Nothing validated skills. Two classes of bug shipped because of that:

1. Silent skill loss. A description: written as an unquoted YAML scalar containing a colon-space is a YAML parse error, not a lint nit. The skills CLI prints one dim ⚠ Skipped … YAML parse error line and then exits 0 — wrong count, install proceeds. So an SDK installs 6 of 7 skills while marketplace.json advertises 7. Live today in deepgram-js-sdk (deepgram-js-text-intelligence) and deepgram-rust-sdk (deepgram-rust-voice-agent); fixes open as deepgram-js-sdk#558 and deepgram-rust-sdk#182. Upstream vercel-labs/skills has this diagnosed in open issues #1094, #1282 and #1525 for 4+ months, so we cannot wait on them.

2. Manifest/reality drift. marketplace.json claimed each SDK ships a deepgram-{lang}-maintaining-sdk skill that exists in no SDK repo. Nothing caught it.

3. Spec staleness. The generated api skill sat ~5 weeks behind the live specs with nothing detecting it. It is behind again right now — see the drift section below.

What is here

• scripts/validate-skills.ts — parses every skills/*/SKILL.md with yaml@2.x, the same parser the installer uses, so results match reality rather than a more lenient parser's opinion. Requires name and description, requires name to equal the directory name, and diffs marketplace.json's local deepgram plugin against the filesystem in both directions. Non-zero exit on any failure.
• .github/workflows/validate-skills.yml — pull_request + push to main.
• .github/workflows/spec-drift.yml — weekly, report-only.

What ci-tools covers vs. what this adds

ci-tools:lint-skills / the check-skill action (skill_lint.py) is wired in — it covers authoring quality this PR does not duplicate: inline-code extraction, allowed-tools permission coverage, model/effort/version frontmatter, draft/ready status, plugin-cache safety. Run against current main it reports 0 errors / 21 warnings / 42 info, so blocking on errors is safe.

It needs a credential before it can actually run, and that is the one thing here that needs a decision. deepgram/ci-tools is a private repository, so the default GITHUB_TOKEN cannot check it out. The first push of this branch proved it:

remote: Repository not found.
fatal: repository 'https://github.com/deepgram/ci-tools/' not found
The process '/usr/bin/git' failed with exit code 128

Rather than red-wall every PR on a missing credential instead of a real skill problem, the job is gated on a CI_TOOLS_READ_TOKEN secret: without it the job writes the gap to the run summary as a warning and passes; with it the job lints and blocks, with no workflow change needed. Someone with repo-admin access needs to provision that secret (a read-scoped PAT or a GitHub App installed on ci-tools) for this lint to be live. The blocking validator job does not depend on it.

It does not catch any of the three bugs above. Verified by injecting them and re-running it:

$ python3 ci-tools/scripts/skill_lint.py audit /work --project   # with bug (a) AND a renamed dir
summary {'errors': 0, 'warnings': 20, 'info': 40}
--- skills/docs/SKILL.md ready          <- unparseable frontmatter, still reported "ready"
--- skills/recipes-renamed/SKILL.md ready   <- name `recipes` != dir `recipes-renamed`
PROJECT FINDINGS:                        <- none

It reads frontmatter with a regex key-match (_has_key) rather than a YAML parser, so an installer-fatal parse error is invisible to it; and its manifest cross-check targets .claude-plugin/plugin.json, which this repo does not have.

check-plugin is deliberately not wired. It hard-fails here, because this is a marketplace repo, not a single-plugin repo:

$ python3 ci-tools/scripts/skill_marketplace.py validate /work
{"ok": false, "errors": [".claude-plugin/plugin.json missing — run scaffold, or author it directly"]}

Adding a plugin.json is a real decision about how this repo is distributed, not a CI change — flagging it rather than doing it.

ci-tools:skill-ci-setup was also evaluated and not adopted: it generates a release workflow (version-stamp → release-notes → commit-back → tag, plus optional catalog-sync), requires plugin.json, and does no validation this PR needs. Standing up release automation for this repo is worth doing, but it is a separate change with its own blast radius.

Proof it works

Passes on current main:

$ bun run scripts/validate-skills.ts
Found 9 skills, 9 valid

All skills valid.
EXIT=0

(a) The exact real-world bug — unquoted description containing a colon-space

Found 9 skills, 8 valid

1 failure(s):
  ✗ skills/voice-agent/SKILL.md: YAML frontmatter failed to parse — Nested mappings are not allowed in compact mappings at line 2, column 14:

description: Route honestly: this crate is the wrong tool for `body: { text }`.
             ^

EXIT=1

Same parser message the installer produces (Nested mappings are not allowed in compact mappings at line 2, column 14) — that is the point of using yaml@2.x rather than something more forgiving.

(b) Directory renamed so name no longer matches

Found 9 skills, 8 valid

1 failure(s):
  ✗ skills/cookbook/SKILL.md: frontmatter `name: recipes` does not match its directory name `cookbook`
EXIT=1

(c) Manifest lists a skill that is missing from disk

Found 8 skills, 8 valid

1 failure(s):
  ✗ .claude-plugin/marketplace.json: plugin "deepgram" lists `./skills/setup-mcp` but skills/setup-mcp/SKILL.md does not exist on disk
EXIT=1

(d) Skill on disk the manifest does not list

Found 10 skills, 10 valid

1 failure(s):
  ✗ .claude-plugin/marketplace.json: skills/self-hosted/SKILL.md exists on disk but plugin "deepgram" does not list `./skills/self-hosted` — it will not install
EXIT=1

(e) marketplace.json is not valid JSON

Found 9 skills, 9 valid

1 failure(s):
  ✗ .claude-plugin/marketplace.json: invalid JSON — JSON Parse error: Unexpected token ','
EXIT=1

All five exit non-zero and name the file. Fixtures were throwaway copies; nothing broken was committed into skills/.

Remote SDK plugin paths

The SDK plugins' skills entries (.agents/skills/*) resolve against remote github sources, so they cannot be checked from a local checkout. Rather than skip them, --remote verifies them against the live repos and, critically, parses their frontmatter too — existence alone would not have caught either production bug.

It runs as a separate continue-on-error job, so a GitHub outage or an in-flight SDK fix never blocks a PR in this repo. Run against the live repos it independently rediscovers both known bugs:

Checking remote SDK plugin skill paths via `gh api`...
  ✗ deepgram/deepgram-js-sdk: .agents/skills/deepgram-js-text-intelligence/SKILL.md — YAML frontmatter failed to parse — Nested mappings are not allowed in compact mappings at line 2, column 14
  ✗ deepgram/deepgram-rust-sdk: .agents/skills/deepgram-rust-voice-agent/SKILL.md — YAML frontmatter failed to parse — Nested mappings are not allowed in compact mappings at line 2, column 14

2 remote SDK plugin skill problem(s) — reported, not blocking.

Because a continue-on-error job green-checks either way and nobody opens a passing job's log, those findings are also written to $GITHUB_STEP_SUMMARY so they are visible on the run page. Shipping another skippable warning would have repeated the original bug.

Spec drift

Weekly (Mondays 13:00 UTC) plus workflow_dispatch. Fetches the specs, regenerates, and compares against the checked-in skills/api/references/. On drift it posts to one rolling spec-drift issue and exits non-zero so the scheduled run is red.

Report only — it never commits regenerated output. Regeneration is a reviewed change; an auto-commit would land unreviewed API copy in a skill agents read as reference.

It does not depend on the concurrent scripts/generate-skills.ts fix. Verified end-to-end against the live specs on today's main, where it correctly reports real drift:

DRIFT DETECTED:
 skills/api/references/agent.md    |  4 +++
 skills/api/references/listen.md   |  4 ++-
 skills/api/references/projects.md | 55 +++++++++++++++++++++++++++++++++++++++
 skills/api/references/speak.md    | 12 ++++-----
 4 files changed, 68 insertions(+), 7 deletions(-)

Two notes:

• Uses the .yml URLs. dpgr.am/openapi.yaml (.yaml) returns the marketing homepage with HTTP 200, and fetch-specs.ts only checks response.ok, so a .yaml typo writes ~509 KB of HTML into specs/ and fails later at parse() with a confusing error. There is an explicit HTML sniff step so a bad fetch reports as a fetch problem.
• Known limitation: the generator overwrites the reference files it emits but does not prune ones it no longer emits, so a reference file for a removed endpoint group reads as no drift. Additions and edits are detected. The job is deliberately tolerant of this rather than asserting the tree is clean; it improves on its own once the prune pass lands.

Gates

Observed on this PR (run 35344542309): Validate skills and manifest passed in 6s, and the remote job found both known SDK bugs in 23s and green-checked as designed.

actionlint (which includes shellcheck on run: blocks) is clean across all three workflows:

verbose: Found total 0 errors in 41 ms for .github/workflows/validate-skills.yml
verbose: Found total 0 errors in 69 ms for .github/workflows/spec-drift.yml
verbose: Found 0 errors in 3 files

Scope

Only three new files. No changes to README.md, .claude-plugin/marketplace.json, AGENTS.md, CHANGELOG.md, CONTRIBUTING.md, scripts/generate-skills.ts, scripts/fetch-specs.ts, or any skills/** file. No package.json change was needed — yaml is already a dependency, and the validator is invoked by path to match the bun run scripts/… convention the README, AGENTS.md and CONTRIBUTING.md already document.

Three things flagged rather than changed: the missing CI_TOOLS_READ_TOKEN secret (gates the ci-tools lint), the missing .claude-plugin/plugin.json (blocks check-plugin), and the generator's missing prune pass (limits the drift check).

[dg-coreylweathers]

Merging last, deliberately, so it validates a consistent tree rather than red-walling the five skill PRs that were still in flight.

Verified on the merged tree myself: Found 14 skills, 14 valid, exit 0.

And I negative-tested it rather than trusting a passing run. Injected the exact production bug — an unquoted description containing `body: { text }` — and got:

✗ skills/zz-fixture/SKILL.md: YAML frontmatter failed to parse
  Nested mappings are not allowed in compact mappings at line 2, column 14
  description: Use when writing code that calls analyze with `body: { text }` or …
               ^
REAL EXIT=1

Byte-identical to what yaml@2 gives the installer, with a caret at the offending column. That is the payoff for using the installer's own parser instead of a lenient one. Also confirmed the present-but-unlisted check fires — which is precisely the gap that let audio-intelligence, text-intelligence, cli, browser-agent and self-hosted sit unregistered in marketplace.json until #18.

(My first attempt reported EXIT=0 and a wrong error message. Both were my own harness bugs — $? capturing tail instead of the validator, and a malformed printf fixture. Worth recording because it is the same trap flagged elsewhere in this batch: verify the gate before trusting the gate.)

Three findings I especially value here:

1. ci-tools:lint-skills does not catch any of the three production bugs, and this was tested by injection rather than assumed: unparseable frontmatter still reports ready, a name/directory mismatch still reports ready, 0 errors, exit 0. Two causes — it reads frontmatter with a regex key-match, never a YAML parser, and its manifest cross-check targets .claude-plugin/plugin.json, which this repo does not have. Wiring it in anyway for what it genuinely does cover, while adding our own checks on top, is the right split.
2. check-plugin correctly not wired. It hard-fails here because this is a marketplace repo, not a single-plugin repo. Adding a plugin.json is a distribution decision, not a CI change.
3. The drift job could not see a NEW reference file in the first draft — git diff --quiet ignores untracked files, so the generator emitting a whole new API domain would have reported clean. Caught by pushing and watching a real run, not by reading. That is exactly the class of bug fix: repair api reference generator (self-hosted routing, WebSocket $refs, Any type stubs) and regenerate #17 just fixed in the generator.

The drift job confirms the api skill is already drifting again (68 insertions across 4 files as of this branch), report-only, never auto-committing, posting to one rolling issue. Its documented limitation — the generator overwrites but does not prune, so a removed endpoint group reads as no drift — is now obsolete: #17 added the prune pass, so it improves on its own.

One thing needs a human with repo-admin: deepgram/ci-tools is private, and the default GITHUB_TOKEN cannot read it. The lint job is gated on a CI_TOOLS_READ_TOKEN secret and currently emits a skip warning and passes, rather than red-walling every PR on a missing credential. Someone needs to provision a read-scoped PAT or a GitHub App installed on ci-tools for that lint to go live. Tracking.