-
-
Notifications
You must be signed in to change notification settings - Fork 9
feat(skill): Add flaky-test-detection skill #61
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
NicolasMassart
merged 4 commits into
main
from
MCWP-473-create-flaky-unit-tests-prevention-and-detection-agent-skill
Jul 2, 2026
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
d32eca9
init skill
NicolasMassart 2cf8fde
Merge branch 'main' into MCWP-473-create-flaky-unit-tests-prevention-…
NicolasMassart c265dc6
fix(gh-analysis): update grep patterns to support .jsx file extension…
NicolasMassart ae3fba6
fix(gh-analysis): update rate calculation precision and add Bash vers…
NicolasMassart File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
113 changes: 113 additions & 0 deletions
113
domains/coding/skills/flaky-test-detection/references/gh-analysis.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,113 @@ | ||
| # Flaky Test Audit — GitHub Actions | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| ```bash | ||
| gh auth status | ||
| git remote get-url origin # must contain MetaMask/metamask-mobile | ||
| ``` | ||
|
|
||
| ## Step 1 — Find the exact unit-test workflow name | ||
|
|
||
| ```bash | ||
| # List recent runs across all workflows — note the name column | ||
| gh run list --repo MetaMask/metamask-mobile --limit 50 | ||
|
|
||
| # Confirm the correct name by filtering (adjust if different from "Unit Tests") | ||
| gh run list --repo MetaMask/metamask-mobile --workflow "Unit Tests" --limit 5 | ||
| ``` | ||
|
|
||
| Use the **exact** workflow name returned by `gh run list` (e.g. `"Unit Tests"`, `"Jest"`, `"CI"`). Substitute it in every `--workflow` flag in the steps below. If the filter returns 0 results, the name is wrong — re-check Step 1. | ||
|
|
||
| ## Step 2 — Collect IDs of failed runs | ||
|
|
||
| Replace `"Unit Tests"` with the name found in Step 1. | ||
|
|
||
| ```bash | ||
| FAILED_RUN_IDS=$(gh run list \ | ||
| --repo MetaMask/metamask-mobile \ | ||
| --workflow "Unit Tests" \ | ||
| --limit 200 \ | ||
| --json databaseId,conclusion \ | ||
| --jq '.[] | select(.conclusion == "failure") | .databaseId') | ||
| ``` | ||
|
|
||
| ## Step 3 — Extract failing test file names | ||
|
|
||
| ```bash | ||
| gh run view <run-id> --repo MetaMask/metamask-mobile --log-failed \ | ||
| | grep -oE 'FAIL .+\.test\.(ts|tsx|js|jsx)' \ | ||
| | sed 's/^FAIL //' | ||
| ``` | ||
|
|
||
| ## Step 4 — Compute per-file failure count and rate | ||
|
|
||
| `TOTAL_RUNS` is all runs in the sampled window (not just failures) — the denominator for rate calculation. It is fetched separately at the end of the script. | ||
|
|
||
| > Requires Bash 4+ (for `declare -A`). macOS ships Bash 3.2 by default — run this with `brew install bash` or via `gh`'s bundled environment, or invoke as `bash script.sh` with a Bash 4+ binary on `PATH`. | ||
|
|
||
| ```bash | ||
| declare -A fail_count | ||
|
NicolasMassart marked this conversation as resolved.
|
||
|
|
||
| for run_id in $FAILED_RUN_IDS; do | ||
| files=$(gh run view "$run_id" --repo MetaMask/metamask-mobile --log-failed 2>/dev/null \ | ||
| | grep -oE 'FAIL .+\.test\.(ts|tsx|js|jsx)' \ | ||
| | sed 's/^FAIL //') | ||
|
Copilot marked this conversation as resolved.
|
||
| for f in $files; do | ||
| fail_count[$f]=$(( ${fail_count[$f]:-0} + 1 )) | ||
| done | ||
| done | ||
|
|
||
| # Count total runs sampled (all runs in the window, not just failures) | ||
| TOTAL_RUNS=$(gh run list \ | ||
| --repo MetaMask/metamask-mobile \ | ||
| --workflow "Unit Tests" \ | ||
| --limit 200 \ | ||
| --json databaseId \ | ||
| --jq 'length') | ||
|
|
||
| # Print: failures total rate% file | ||
| for f in "${!fail_count[@]}"; do | ||
| count=${fail_count[$f]} | ||
| rate=$(echo "scale=2; $count * 100 / $TOTAL_RUNS" | bc) | ||
| echo "$count $TOTAL_RUNS ${rate}% $f" | ||
| done | sort -rn | ||
| ``` | ||
|
|
||
| > Rate limit: ~5,000 gh API requests/hour. Add `sleep 1` between loop iterations for large histories (>100 runs). | ||
|
|
||
| ## Step 5 — Ranked output format | ||
|
|
||
| Columns: rank, test file path, failure count, total runs sampled, failure rate, pattern category (from symptom mapping below). | ||
|
|
||
| ``` | ||
| Rank | Test file | Failures | Runs | Rate | Category | ||
| -----|-------------------------------------------------------|---------:|-----:|------:|--------- | ||
| 1 | app/components/Views/BankDetails/BankDetails.test.tsx | 12 | 50 | 24 % | J1 | ||
| 2 | app/util/transactions/transactions.test.ts | 9 | 50 | 18 % | J3 | ||
| ``` | ||
|
|
||
| Assign the category by matching the failure log symptoms in the table below, then open the pattern section in the skill for the fix. | ||
|
|
||
| ## Symptom → category mapping | ||
|
|
||
| | Symptom in failed log | Category | | ||
| |---|---| | ||
| | `TypeError: terminated` / `SocketError: other side closed` | J1 | | ||
| | Timer / timeout mismatch | J2 | | ||
| | `toHaveBeenCalledTimes` wrong count | J3 | | ||
| | `waitFor` never resolved | J4 or J8 | | ||
| | `Cannot read property of undefined` from store selector | J5 | | ||
| | Test timed out (5 s default) | J6 | | ||
| | Assertion on `Date` or random value fails | J7 | | ||
| | Test hangs indefinitely under fake timers | J8 | | ||
| | Test passes in isolation but fails in suite | J9 | | ||
| | Unexpected function called / wrong return value | J10 | | ||
|
|
||
| ## Thresholds | ||
|
|
||
| | Rate | Action | | ||
| |------|--------| | ||
| | ≥ 10 % | Actively flaky — fix before merging new tests in that file | | ||
| | 5–10 % | At-risk — review for patterns in its category | | ||
| | < 5 % | Monitor | | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.