Skip to content

test(mcp): opt-in live output schema conformance tests - #53

Merged
Kikobeats merged 1 commit into
masterfrom
feat/mcp-live-schema-tests
Sep 13, 2026
Merged

test(mcp): opt-in live output schema conformance tests#53
Kikobeats merged 1 commit into
masterfrom
feat/mcp-live-schema-tests

Conversation

@Kikobeats

@Kikobeats Kikobeats commented Sep 12, 2026

Copy link
Copy Markdown
Member

What

A new test/live.test.js that invokes real tools against the free Microlink endpoint and validates structuredContent against each tool's declared outputSchema.

Motivation: every output schema correction so far (logo/video/audio null, markdown/html/text null, embed null, lighthouse string output) came from real API behavior that stubbed unit tests could not catch. These tests make that drift visible on demand.

Coverage

One tool per schema family: metadata, markdown, logo, links, screenshot, embed, technologies, extract. Skips slow/keyed tools (search, function, lighthouse, pdf).

Opt-in

Every test is skipped unless LIVE=1, so CI and local runs are untouched (default suite: 101 pass, 8 skipped). New script: pnpm test:live.

Verified just now against the live free endpoint: 8/8 pass.


Note

Low Risk
Test-only, opt-in changes with no production or CI behavior unless LIVE=1 is set.

Overview
Adds opt-in live conformance tests for the MCP package so real Microlink API responses can be checked against each tool’s declared outputSchema.

A new test/live.test.js registers all tools via the same tools() hook, invokes handlers against the free endpoint, and uses Zod to validate structuredContent (with a 120s timeout). Eight cases cover one representative tool per schema family (metadata, markdown, logo, links, screenshot, embed, technologies, extract); everything is skipped unless LIVE=1, so default pnpm test and CI stay unchanged.

package.json gains pnpm test:live (LIVE=1 node --test test/live.test.js) alongside the existing test script.

Reviewed by Cursor Bugbot for commit 96c8881. Bugbot is set up for automated code reviews on this repo. Configure here.

Summary by CodeRabbit

  • Tests
    • Added a command for running the live test suite with live-mode settings enabled.

Invoke real tools against the free Microlink endpoint and validate
structuredContent against each tool's declared outputSchema. Catches
contract drift between schemas and the real API that stubbed unit
tests cannot (the null media/content/embed cases all came from live
behavior).

Skipped unless LIVE=1, so CI and local runs are unaffected. Run them
with pnpm test:live.

Signed-off-by: Kiko Beats <2096101+Kikobeats@users.noreply.github.com>
@coderabbitai

coderabbitai Bot commented Sep 12, 2026

Copy link
Copy Markdown

Review Change StackReview Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: d21da980-8292-482e-b3cd-d025a9322faa

📥 Commits

Reviewing files that changed from the base of the PR and between f2d20e6 and 96c8881.

📒 Files selected for processing (2)
  • packages/mcp/package.json
  • packages/mcp/test/live.test.js

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.


📝 Walkthrough

Walkthrough

The MCP package adds a test:live script. The script enables LIVE and runs test/live.test.js with Node's test runner.

Changes

MCP live test execution

Layer / File(s) Summary
Add live test script
packages/mcp/package.json
Adds test:live, which runs node --test test/live.test.js with LIVE enabled.

Priority: ⬇️ Low

Estimated code review effort: 1 (Trivial) | ~5 minutes

Change: Other

Merge Risk: ⚪ Minimal · up to 96c88

This adds an opt-in command for live schema tests without changing default tests or production behavior. No merge-blocking risk is evident.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 1 files. (1 skipped: 1 … Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the main change: adding opt-in live MCP output schema conformance tests.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 1 files. (1 skipped: 1 unsupported.)

  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/mcp-live-schema-tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coveralls

Copy link
Copy Markdown

Coverage Report for CI Build 34702088029

Warning

No base build found for commit f2d20e6 on master.
Coverage changes can't be calculated without a base build.
If a base build is processing, this comment will update automatically when it completes.

Coverage: 79.358%

Details

  • Patch coverage: No coverable lines changed in this PR.

Uncovered Changes

No uncovered changes found.

Coverage Regressions

Requires a base build to compare against. How to fix this →


Coverage Stats

Coverage Status
Relevant Lines: 5152
Covered Lines: 4098
Line Coverage: 79.54%
Relevant Branches: 865
Covered Branches: 677
Branch Coverage: 78.27%
Branches in Coverage %: Yes
Coverage Strength: 25.33 hits per line

💛 - Coveralls

@Kikobeats

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Sep 12, 2026

Copy link
Copy Markdown
⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@Kikobeats

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Sep 13, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@Kikobeats
Kikobeats merged commit 2fa3269 into master Sep 13, 2026
9 checks passed
@Kikobeats
Kikobeats deleted the feat/mcp-live-schema-tests branch September 13, 2026 07:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants