WEB-1238: [CI/CD] Surface retry outcomes from Playwright E2E as a flake signal

[devvaansh]

Description

playwright.config.ts retries failed tests twice in CI (retries: process.env.CI ? 2 : 0). That keeps PRs unblocked on transient noise, but it means a test can fail and then pass on a retry inside a run that ends green.

The github reporter already names such tests — its 🎭 Playwright Run Summary annotation has an N flaky line — but the signal is easy to miss and hard to use:

• The failed attempt of a flaky test is annotated as a red error, indistinguishable from a real failure, on a run that passed.
• Nothing is written to the job summary, so the run page shows no pass / fail / flaky breakdown without digging through annotations.
• Nothing is kept in machine-readable form, so flaky counts can't be compared run-over-run or against the nightly canary (WEB-1184: [CI/CD] Add a nightly Fineract drift canary #3917).

This adds:

1. The json reporter to npm run playwright:ci, writing to playwright-report/results.json via PLAYWRIGHT_JSON_OUTPUT_FILE. It sits inside playwright-report/, so the existing if: always() upload keeps it on every run — no new artifact.
2. scripts/summarize-flaky-tests.js, which reads that report and writes to $GITHUB_STEP_SUMMARY:
   • a counts line on every run, e.g. 0 flaky · 236 passed · 0 failed · 5 skipped, so silence is never the signal;
   • when there are flaky tests, a table with each test's repo-relative location and title path, its project, the attempt count, and the first line of its first failure.
3. A Summarize flaky tests step right after Run Playwright tests, with if: always() and continue-on-error: true, so it reports on any run but can never change the job's result.

Illustrative output — this repo's real test list, with two tests forced flaky:

2 flaky · 0 passed · 0 failed · 239 skipped

Test	Project	Attempts	First failure
playwright/factories/_shared.spec.ts:21 › resolveProductId › reads the id spelling returned by ensure*Product	unit	2	TimeoutError: locator.click: Timeout 30000ms exceeded.
playwright/tests/authenticated-smoke.spec.ts:23 › Authenticated Smoke Tests › should load dashboard without login redirect	chromium	2	TimeoutError: locator.click: Timeout 30000ms exceeded.

Design notes worth reviewing:

• json has to stay after html in playwright:ci. Playwright calls each reporter's onEnd in list order, and the html reporter clears playwright-report/ when it finishes. With html,github,json the JSON file is written after that and survives; with json first it is deleted. A comment next to PLAYWRIGHT_JSON_OUTPUT_FILE records this.
• The output path comes from an env var, not the config. --reporter on the CLI replaces the config's reporter list, so an outputFile in playwright.config.ts would not apply. Without the variable, the JSON goes to stdout.
• If the test step never finishes (for example the stack fails to come up), there is no report; the summary says so instead of the step failing.
• Retry and worker settings are unchanged, and no check is added or made required.

Current baseline, for context: across 159 completed Playwright E2E runs between 2026-09-02 and 2026-09-11, 2 runs had one flaky test each — clients/create-client.spec.ts:95 and close-client.spec.ts:128, both locator timeouts. So 0 flaky is the expected line on nearly every run; the point is that it stays measured rather than assumed.

Note for reviewers: .gitignore line 76 (scripts/, added together with docs/ in b23dcbc) matches this directory even though every script in it is tracked — render-amplify-env.js was added after the rule. The new script is tracked the same way; the ignore rule itself is left untouched here.

Related issues and discussion

WEB-1238

Related: #3917 (WEB-1184) — the nightly drift canary this gives a flaky-test baseline to.

Screenshots, if any

N/A — the job summary of this PR's own Playwright E2E run shows the output.

Checklist

Please make sure these boxes are checked before submitting your pull request - thanks!

• If you have multiple commits please combine them into one commit by squashing them.

• Read and understood the contribution guidelines at web-app/.github/CONTRIBUTING.md.

Summary by CodeRabbit

• New Features

  • Added automated summaries of flaky Playwright tests, including affected projects, test locations, retry attempts, and first-failure details.
  • Summaries are published to the GitHub Actions job summary when available, or displayed in the command output.

• Testing

  • Playwright CI runs now generate a JSON report alongside existing HTML and GitHub reports, enabling automated flaky-test analysis.

[coderabbitai]

Note

.coderabbit.yaml has unrecognized properties

CodeRabbit is using all valid settings from your configuration. Unrecognized properties (listed below) have been ignored and may indicate typos or deprecated fields that can be removed.

⚠️ Parsing warnings (1)

Validation error: Unrecognized key: "pre_merge_checks"

⚙️ Configuration instructions

• Please see the configuration documentation for more information.
• You can also validate your configuration using the online YAML validator.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: ffa99431-6ab0-4392-86ed-0cf6f0c7d44c

📥 Commits

Reviewing files that changed from the base of the PR and between 1009c9b and 66cf5b6.

⛔ Files ignored due to path filters (1)

• .github/workflows/playwright.yml is excluded by !.github/** and included by **/*

📒 Files selected for processing (2)

• package.json
• scripts/summarize-flaky-tests.js

Included review availability: Your plan provides up to 2 included reviews per hour; 1 remains after this review.

---

Walkthrough

Playwright CI now emits a JSON report. A new Node.js script reads the report, identifies flaky tests, sanitizes failure details, and writes a Markdown summary to the GitHub step summary or stdout.

Changes

Playwright flaky reporting

Layer / File(s)	Summary
Generate Playwright JSON reports package.json, scripts/summarize-flaky-tests.js	The CI Playwright command adds the json reporter. The summary script defines report-processing configuration and dependencies.
Collect and sanitize flaky tests scripts/summarize-flaky-tests.js	The script recursively collects flaky tests, normalizes report paths, and sanitizes first-failure messages for Markdown output.
Render and publish the summary scripts/summarize-flaky-tests.js	The script renders test statistics and flaky-test details. It validates the report argument and writes output to GITHUB_STEP_SUMMARY or stdout.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Feature

Suggested reviewers: alberto-art3ch

Sequence Diagram(s)

sequenceDiagram
  participant PlaywrightCI
  participant JSONReport
  participant SummaryScript
  participant GitHubStepSummary
  PlaywrightCI->>JSONReport: Generate Playwright test results
  SummaryScript->>JSONReport: Read report
  SummaryScript->>SummaryScript: Find and sanitize flaky tests
  SummaryScript->>GitHubStepSummary: Write Markdown summary

Loading

Merge Risk: ⚪ Minimal · up to 3c278

The CI reporting change has no confirmed merge-blocking risk.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 5 functions across 1 files. (1 skipped: 1 …	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: reporting Playwright retry outcomes as a signal for flaky tests in CI/CD.

Full details: Docstring Coverage

Explanation

Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 5 functions across 1 files. (1 skipped: 1 unsupported.)

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}