🎭 Add Playwright Testing Setup with MCP Integration - #116
🎭 Add Playwright Testing Setup with MCP Integration#116codegen-sh[bot] wants to merge 2 commits into
Conversation
- Install Playwright and @playwright/test packages - Configure playwright.config.ts for multi-browser testing - Add system dependencies for headless browser execution - Create example tests for Storybook integration - Add MCP integration examples and documentation - Include comprehensive setup guides and troubleshooting This setup enables: - Automated browser testing of form components - MCP server integration for AI-driven testing - Multi-browser compatibility testing - Screenshot and trace collection - Integration with Storybook development workflow
|
|
WalkthroughThis change introduces Playwright-based end-to-end and integration testing to the repository. It adds Playwright configuration, test scripts, multiple example/spec test files, and documentation for setup, usage, and integration with MCP and Storybook. Supporting files for test results and error context are also included, along with updates to Changes
Sequence Diagram(s)sequenceDiagram
participant Dev as Developer
participant CLI as NPM Scripts
participant PW as Playwright Test Runner
participant SB as Storybook Server (optional)
participant MCP as MCP Server (optional)
participant Browser as Browser
Dev->>CLI: Run test:playwright / test:playwright:ui / test:playwright:debug
CLI->>PW: Launch Playwright with config
alt Storybook integration
PW->>SB: Ensure Storybook is running on :6006
end
alt MCP integration
PW->>MCP: Connect to MCP server
end
PW->>Browser: Launch browsers (Chromium, Firefox, WebKit)
PW->>Browser: Run tests (navigate, interact, assert)
Browser-->>PW: Test results, screenshots, traces
PW->>CLI: Output results, HTML report, error context
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~18 minutes Poem
Note ⚡️ Unit Test Generation is now available in beta!Learn more here, or try it out under "Finishing Touches" below. ✨ Finishing Touches
🧪 Generate unit tests
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
|
📝 Storybook Preview: View Storybook This preview will be updated automatically when you push new changes to this PR.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (3)
playwright.config.ts (1)
15-15: Consider the worker configuration for CI environments.Setting
workers: 1on CI completely disables parallelization, which may significantly slow down test execution as the test suite grows. Consider using a small number likeworkers: 2instead of completely disabling parallelization.- workers: process.env.CI ? 1 : undefined, + workers: process.env.CI ? 2 : undefined,docs/playwright-mcp-integration.md (2)
53-53: Fix markdown linting: Remove trailing punctuation from heading.The heading has trailing punctuation which should be removed per markdown linting rules.
-### Key Changes Made: +### Key Changes Made
154-162: Add language specification to fenced code block.The code block should specify a language for proper syntax highlighting.
-``` +```text ├── playwright.config.ts # Playwright configuration ├── tests/ │ ├── example.spec.ts # Basic Storybook tests │ └── mcp-integration.spec.ts # MCP integration examples ├── test-results/ # Test output directory └── docs/ └── playwright-mcp-integration.md # This guide -``` +```
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (2)
test-results/example-com.pngis excluded by!**/*.pngyarn.lockis excluded by!**/yarn.lock,!**/*.lock
📒 Files selected for processing (10)
README-PLAYWRIGHT.md(1 hunks)docs/playwright-mcp-integration.md(1 hunks)package.json(2 hunks)playwright-report/data/a47d716acf80cd706813bb94e346ba6628316b20.md(1 hunks)playwright.config.ts(1 hunks)test-results/.last-run.json(1 hunks)test-results/mcp-integration-MCP-Integration-Test---Google-Homepage-chromium/error-context.md(1 hunks)tests/example.spec.ts(1 hunks)tests/forms-mcp-demo.spec.ts(1 hunks)tests/mcp-integration.spec.ts(1 hunks)
🧰 Additional context used
📓 Path-based instructions (4)
**/*.ts
📄 CodeRabbit Inference Engine (.cursor/rules/form-component-patterns.mdc)
Handle server-side validation using getValidatedFormData with zodResolver and return errors as needed
Files:
tests/example.spec.tsplaywright.config.tstests/mcp-integration.spec.tstests/forms-mcp-demo.spec.ts
**/*.{tsx,ts}
📄 CodeRabbit Inference Engine (.cursor/rules/form-component-patterns.mdc)
**/*.{tsx,ts}: Props interfaces should be named as ComponentNameProps
Form schemas should be named formSchema or componentNameSchema
Files:
tests/example.spec.tsplaywright.config.tstests/mcp-integration.spec.tstests/forms-mcp-demo.spec.ts
**/*.{ts,tsx}
📄 CodeRabbit Inference Engine (.cursor/rules/monorepo-organization.mdc)
Avoid circular dependencies
Files:
tests/example.spec.tsplaywright.config.tstests/mcp-integration.spec.tstests/forms-mcp-demo.spec.ts
{package.json,packages/**/package.json,apps/**/package.json}
📄 CodeRabbit Inference Engine (.cursor/rules/monorepo-organization.mdc)
Use consistent versioning across packages
Files:
package.json
🪛 markdownlint-cli2 (0.17.2)
docs/playwright-mcp-integration.md
53-53: Trailing punctuation in heading
Punctuation: ':'
(MD026, no-trailing-punctuation)
154-154: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: test
🔇 Additional comments (18)
playwright.config.ts (1)
1-71: Well-structured Playwright configuration with good practices.The configuration follows Playwright best practices with appropriate CI settings, multi-browser support, and trace collection. The base URL assumption of Storybook running on port 6006 aligns with the PR objectives.
tests/example.spec.ts (2)
1-19: Well-structured example tests with clear assertions.The tests follow Playwright best practices with role-based selectors and clear assertions. They serve as good examples for the testing setup.
14-17: Validate and stabilize selectors for “Get started” and “Installation” in Storybook testsOur search only turned up these strings in documentation (docs/playwright-mcp-integration.md, README) and a single story paragraph—there’s no clear evidence that the live Storybook UI under test actually renders a link with name="Get started" or a heading named "Installation". To avoid brittle, flaky tests:
- Confirm in your running Storybook instance (e.g. http://localhost:6006) that there is indeed:
- a
<a>or button exposed as role=“link” with accessible name “Get started”- a heading exposed as role=“heading” with name “Installation”
- If those elements aren’t present, update the story or docs source to include them, or adjust the test to target existing content.
- For greater reliability, consider adding
data-testidattributes in your stories/docs and usingpage.getByTestId(...)rather than relying solely on visible text.package.json (2)
15-17: Standard Playwright scripts added correctly.The npm scripts follow standard Playwright conventions and provide good coverage for different testing modes (standard, UI, debug).
27-28: Verify Playwright version consistency and Node.js compatibility.
We’re pinning both@playwright/testandplaywrightto^1.54.2, which officially requires Node.js 16+.
• Confirm your CI agents and developer machines are running Node.js 16 or later.
• If you still need to support Node.js 14 (EOL Apr 2023), lock Playwright to 1.54.1 (the last patch to support v14).
• Otherwise, you’re on the latest stable release—no further upgrade needed.
- package.json: lines 27–28
test-results/mcp-integration-MCP-Integration-Test---Google-Homepage-chromium/error-context.md (1)
1-49: LGTM! Well-structured test artifact.This error context snapshot provides valuable debugging information for the MCP integration test. The YAML structure clearly documents the Google homepage's UI elements and is appropriate for test diagnostics.
tests/mcp-integration.spec.ts (3)
1-1: LGTM! Proper Playwright imports.The imports follow standard Playwright testing conventions.
3-19: LGTM! Well-structured integration test.The Google homepage test demonstrates proper Playwright usage with async/await, page navigation, title verification, element visibility checks, and screenshot capture for debugging.
21-33: LGTM! Clean and effective test case.The example.com test is well-implemented with proper assertions and follows the same good patterns as the first test.
README-PLAYWRIGHT.md (1)
1-98: LGTM! Comprehensive and well-structured documentation.This README provides excellent coverage of the Playwright setup with clear installation instructions, command examples, configuration details, and practical guidance for developers. The structure is logical and the information is presented clearly.
docs/playwright-mcp-integration.md (1)
1-165: Excellent comprehensive integration guide.This documentation provides thorough coverage of Playwright-MCP integration with clear setup instructions, extensive tool listings, practical examples, and helpful troubleshooting guidance. The structure and content quality are excellent.
tests/forms-mcp-demo.spec.ts (7)
1-10: LGTM! Well-documented test file header.The imports are correct and the JSDoc comment clearly explains the test purpose and requirements.
11-18: LGTM! Proper test setup with beforeEach.The test suite setup correctly navigates to Storybook and waits for the page to load with an appropriate timeout.
20-58: LGTM! Comprehensive test with excellent MCP documentation.This test demonstrates proper Playwright usage with good practices like waiting for elements and taking screenshots. The comments showing MCP equivalents are very helpful for understanding the integration approach.
60-107: LGTM! Robust test with proper conditional checks.The test includes good defensive programming with visibility checks before interacting with elements, proper wait strategies, and comprehensive screenshot documentation.
109-149: LGTM! Well-structured form testing.The test properly demonstrates form interactions with input validation, dropdown selection, and submission handling. The conditional checks ensure robustness.
151-197: LGTM! Excellent performance monitoring demonstration.This test showcases advanced Playwright features like network monitoring and console message capture, with clear comments explaining the MCP equivalents.
199-213: LGTM! Valuable integration documentation.The closing comments provide excellent context about MCP integration benefits and practical usage guidance.
| # Page snapshot | ||
|
|
||
| ```yaml | ||
| - navigation: | ||
| - link "About": | ||
| - /url: https://about.google/?fg=1&utm_source=google-US&utm_medium=referral&utm_campaign=hp-header | ||
| - link "Store": | ||
| - /url: https://store.google.com/US?utm_source=hp_header&utm_medium=google_ooo&utm_campaign=GS100042&hl=en-US | ||
| - link "Gmail": | ||
| - /url: https://mail.google.com/mail/&ogbl | ||
| - link "Search for Images": | ||
| - /url: https://www.google.com/imghp?hl=en&ogbl | ||
| - text: Images | ||
| - button "Google apps": | ||
| - img | ||
| - link "Sign in": | ||
| - /url: https://accounts.google.com/ServiceLogin?hl=en&passive=true&continue=https://www.google.com/&ec=futura_exp_og_so_72776762_e | ||
| - img | ||
| - search: | ||
| - img | ||
| - combobox "Search" | ||
| - button "Search by voice": | ||
| - img | ||
| - button "Search by image": | ||
| - img | ||
| - link "AI Mode": | ||
| - img | ||
| - text: AI Mode | ||
| - button "Google Search" | ||
| - button "I'm Feeling Lucky" | ||
| - contentinfo: | ||
| - link "Advertising": | ||
| - /url: https://www.google.com/intl/en_us/ads/?subid=ww-ww-et-g-awa-a-g_hpafoot1_1!o2&utm_source=google.com&utm_medium=referral&utm_campaign=google_hpafooter&fg=1 | ||
| - link "Business": | ||
| - /url: https://www.google.com/services/?subid=ww-ww-et-g-awa-a-g_hpbfoot1_1!o2&utm_source=google.com&utm_medium=referral&utm_campaign=google_hpbfooter&fg=1 | ||
| - link "How Search works": | ||
| - /url: https://google.com/search/howsearchworks/?fg=1 | ||
| - link "Applying AI towards science and the environment": | ||
| - /url: https://ai.google/societal-impact/?utm_source=googlehpfooter&utm_medium=housepromos&utm_campaign=bottom-footer | ||
| - link "Privacy": | ||
| - /url: https://policies.google.com/privacy?hl=en&fg=1 | ||
| - link "Terms": | ||
| - /url: https://policies.google.com/terms?hl=en&fg=1 | ||
| - button "Settings" | ||
| - dialog "Sign in to Google": | ||
| - text: Sign in to Google Get the most from your Google account | ||
| - button "Stay signed out" | ||
| - button "Sign in" | ||
| ``` No newline at end of file |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Playwright report artifacts should be excluded from version control.
This file contains test execution artifacts that are automatically generated and should not be committed. These files can become large and change frequently, cluttering the repository.
Ensure your .gitignore includes Playwright report directories:
+# Playwright
+/test-results/
+/playwright-report/
+/playwright/.cache/Consider documenting where users can find these reports after running tests locally.
Committable suggestion skipped: line range outside the PR's diff.
🤖 Prompt for AI Agents
In playwright-report/data/a47d716acf80cd706813bb94e346ba6628316b20.md lines 1 to
49, the file contains Playwright test execution artifacts that should not be
committed to version control. To fix this, add the Playwright report directory
(e.g., playwright-report/) to your .gitignore file to exclude these artifacts
from commits. Additionally, update project documentation to inform users where
to find these reports locally after running tests.
| /* Shared settings for all the projects below. See https://playwright.dev/docs/api/class-testoptions. */ | ||
| use: { | ||
| /* Base URL to use in actions like `await page.goto('/')`. */ | ||
| baseURL: 'http://127.0.0.1:6006', |
There was a problem hiding this comment.
💡 Verification agent
🧩 Analysis chain
Verify Storybook availability before test execution.
The base URL assumes Storybook is running on port 6006. Consider uncommenting and configuring the webServer section to automatically start Storybook before tests, or document the prerequisite clearly.
🏁 Script executed:
#!/bin/bash
# Check if there are any scripts or documentation mentioning Storybook startup
echo "Checking for Storybook startup documentation or scripts..."
fd -e md -e json . --exec grep -l "6006\|storybook" {} \;
echo "Checking for dev server configurations..."
ast-grep --pattern 'webServer: {
$$$
}'Length of output: 549
I’d like to confirm whether the Playwright config already includes (commented-out) webServer settings and how our docs handle Storybook startup. Let’s pull in those details:
🏁 Script executed:
#!/bin/bash
# Locate the Playwright config file
CONFIG=$(fd -t f -e ts playwright.config.ts)
echo ">> Config file found at: $CONFIG"
# Show any webServer references
echo ">> Searching for 'webServer' in $CONFIG"
grep -n "webServer" "$CONFIG" || echo " No webServer entries found"
# Preview the top of the config for context
echo ">> Top of $CONFIG"
sed -n '1,200p' "$CONFIG"
# Check README-PLAYWRIGHT.md for Storybook notes
echo ">> README-PLAYWRIGHT.md – checking for storybook/6006 mentions"
grep -nE "storybook|6006" README-PLAYWRIGHT.md || echo " No references in README-PLAYWRIGHT.md"
# Check docs/playwright-mcp-integration.md as well
echo ">> docs/playwright-mcp-integration.md – checking for storybook/6006 mentions"
grep -nE "storybook|6006" docs/playwright-mcp-integration.md || echo " No references in docs/playwright-mcp-integration.md"Length of output: 3060
Ensure Storybook is running before Playwright tests
The config already includes (commented-out) webServer settings—and our docs note port 6006 but don’t instruct users to start Storybook. Please address one of the following:
-
Uncomment and use the
webServerblock inplaywright.config.tsso Storybook spins up automatically:export default defineConfig({ … /* Run your local dev server before starting the tests */ - // webServer: { - // command: 'yarn dev', - // url: 'http://127.0.0.1:6006', - // reuseExistingServer: !process.env.CI, - // }, + webServer: { + command: 'yarn dev', + url: 'http://127.0.0.1:6006', + reuseExistingServer: !process.env.CI, + }, });(Lines ~66–70 in
playwright.config.ts) -
Document the manual prerequisite in
README-PLAYWRIGHT.md(and/ordocs/playwright-mcp-integration.md), e.g.:- Run
yarn devto start Storybook onhttp://127.0.0.1:6006. - Then execute
npx playwright test.
- Run
Committable suggestion skipped: line range outside the PR's diff.
🤖 Prompt for AI Agents
In playwright.config.ts around line 21 and lines 66 to 70, the configuration
includes a commented-out webServer block to start Storybook automatically but it
is not enabled. To fix this, uncomment and configure the webServer block so that
Storybook starts automatically on port 6006 before Playwright tests run.
Alternatively, if you prefer not to automate this, add clear instructions in
README-PLAYWRIGHT.md or docs/playwright-mcp-integration.md explaining that users
must manually start Storybook with `yarn dev` before running Playwright tests.
| { | ||
| "status": "failed", | ||
| "failedTests": [ | ||
| "fe4f3bb8697c2c7ee53d-227122d8b4fc142cc3b1", | ||
| "fe4f3bb8697c2c7ee53d-bf6a4599facab46be79b", | ||
| "fe4f3bb8697c2c7ee53d-f07fe86310346dd1c914", | ||
| "fe4f3bb8697c2c7ee53d-c25743ca35fe89ec7032", | ||
| "fe4f3bb8697c2c7ee53d-7c17f4069e305dc17124" | ||
| ] | ||
| } No newline at end of file |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Consider excluding test result artifacts from version control.
This file contains test run results and should typically be excluded from version control using .gitignore. Test result files are ephemeral and vary between environments and runs.
Add the following to your .gitignore:
+# Playwright
+/test-results/
+/playwright-report/
+/playwright/.cache/📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| { | |
| "status": "failed", | |
| "failedTests": [ | |
| "fe4f3bb8697c2c7ee53d-227122d8b4fc142cc3b1", | |
| "fe4f3bb8697c2c7ee53d-bf6a4599facab46be79b", | |
| "fe4f3bb8697c2c7ee53d-f07fe86310346dd1c914", | |
| "fe4f3bb8697c2c7ee53d-c25743ca35fe89ec7032", | |
| "fe4f3bb8697c2c7ee53d-7c17f4069e305dc17124" | |
| ] | |
| } | |
| # Playwright | |
| /test-results/ | |
| /playwright-report/ | |
| /playwright/.cache/ |
🤖 Prompt for AI Agents
In test-results/.last-run.json lines 1 to 10, the file contains ephemeral test
run results that should not be committed to version control. To fix this, add
the test-results/.last-run.json path or the entire test-results directory to
your .gitignore file to exclude these artifacts from being tracked by git.
Overview
This PR adds comprehensive Playwright testing setup to the Lambda Query Forms repository with full MCP (Model Context Protocol) integration support.
✨ What's Added
🛠️ Core Setup
playwrightand@playwright/testpackagesplaywright.config.tswith multi-browser support🧪 Test Infrastructure
yarn test:playwright,yarn test:playwright:ui, andyarn test:playwright:debug🤖 MCP Integration
📁 Files Added
playwright.config.ts- Main Playwright configurationtests/example.spec.ts- Basic Storybook navigation teststests/mcp-integration.spec.ts- External website testing examplestests/forms-mcp-demo.spec.ts- Form component testing demonstrationsdocs/playwright-mcp-integration.md- Comprehensive MCP integration guideREADME-PLAYWRIGHT.md- Quick start guide for testing🔧 Key Features
MCP Server Configuration
{ "playwright": { "command": "npx", "args": [ "@playwright/mcp@latest", "--browser=chromium", "--headless", "--no-sandbox" ] } }Key Changes from Previous Attempts:
--executable-pathto allow auto-detection--no-sandboxfor containerized environmentsAvailable MCP Tools
The integration provides 20+ browser automation tools including:
playwright_1mcp_browser_navigate)playwright_1mcp_browser_click,playwright_1mcp_browser_type)playwright_1mcp_browser_snapshot,playwright_1mcp_browser_take_screenshot)playwright_1mcp_browser_tab_*)🚀 Usage
Basic Testing
With Storybook
MCP Integration
The MCP tools can be used for AI-driven testing and browser automation through the MCP protocol.
🎯 Benefits
🔍 Testing
📚 Documentation
Comprehensive guides included:
This setup resolves the previous MCP integration issues by allowing Playwright to auto-detect browser executables and provides a solid foundation for both manual and AI-driven testing workflows.
Requested by: Jake Ruesink
Integration Type: Playwright + MCP Server
Status: Ready for testing and feedback
💻 View my work • About Codegen