feat(skill): Add flaky-test-detection skill

[NicolasMassart]

Description

Adds a new flaky-test-detection skill for MetaMask Mobile that detects, prevents, and fixes flaky Jest unit tests. It provides a review mode for scanning changed test files against known flakiness risk patterns (async, timing, isolation, mock, state), and an audit mode that queries GitHub Actions run history to rank historically flaky tests by failure rate.

Type of Change

• New skill

Skill Details (if adding a new skill)

Provider Name: MetaMask
Skill Name: flaky-test-detection
Brief Description: Detect, prevent, and fix flaky Jest unit tests in MetaMask Mobile before they reach CI, via PR/diff review against a risk-pattern table and a GitHub Actions history audit mode.

Checklist

• I have read the CONTRIBUTING.md guidelines
• My skill follows the SKILL_TEMPLATE.md format
• I have tested this skill with an AI agent
• My skill does not contain any secrets, private keys, or sensitive data
• I have added appropriate documentation
• My changes don't break existing skills

Testing

Ran the skill against MetaMask Mobile PR diffs and CI run history to validate both review mode (pattern classification) and audit mode (ranked flaky test output).

Example result from running on Metamask Mobile repo:

---

Flaky test audit — Jest unit tests (ci.yml, last 150 CI runs)

Of 150 recent ci.yml runs, 31 failed overall; only 3 failed due to actual Jest failures (the rest were infra: yarn install retries, build issues, etc.). All 3 are confirmed flaky — same commit passed on a separate re-run.

Rank	Test file	Failures	Runs sampled	Rate	Category
1	app/components/UI/Tokens/TokenList/TokenListItem/TokenListItem.test.tsx	1	150	<1%	J9/J10 (see below)
1	app/components/Views/confirmations/components/gas/selected-gas-fee-token/selected-gas-fee-token.test.tsx	1	150	<1%	J9/J10
1	app/components/Views/confirmations/components/info/contract-deployment/contract-deployment.test.tsx	1	150	<1%	J9/J10
2	app/selectors/assets/balances.test.ts	1	150	<1%	J1/J3
2	app/components/Views/confirmations/components/confirm/confirm-component.test.tsx	1	150	<1%	J1/J3
2	app/components/Views/confirmations/hooks/gas/useGasFeeToken.test.ts	1	150	<1%	J1/J3
2	app/components/Views/confirmations/components/gas/gas-fee-token-toast/gas-fee-token-toast.test.tsx	1	150	<1%	J1/J3
2	app/components/Views/confirmations/components/gas/gas-fee-token-list-item/gas-fee-token-list-item.test.tsx	1	150	<1%	J1/J3
3	app/core/Engine/Engine.test.ts	1	150	<1%	untriaged (unrelated PR retry)

Individually every file's rate is below the 5% "monitor" threshold, but the two ranked incidents matter more than the per-file rate:

Incident 1 — feat/stellar/ramp-onramp-support PR, commit 9953242: run 28582173232 failed, run 28582184123 on the exact same commit passed. All 3 unrelated files threw TypeError: selectAccountByScope is not a function in the same shard.

Incident 2 — same PR, earlier commit ff7051b: run 28577001112 — 5 unrelated files (selectors, a confirm hook, gas-fee-token components) all threw the identical TypeError: Cannot read properties of undefined (reading 'some'). In the same job log there's also ReferenceError: You are trying to import a file after the Jest environment has been torn down from DeeplinkProtocolService.test.ts right before the cascade starts.

Both incidents share a signature: several unrelated test files in the same worker/shard fail simultaneously with the identical generic error, and the same commit is green elsewhere. That's not per-file flakiness — it's cross-test contamination inside a shared Jest worker, consistent with skill categories J1 (dangling async work bleeding past test teardown — the "torn down" error is the smoking gun) and J9/J10 (module-level mock state not reset between files in the same worker, causing selectAccountByScope/array-selector mocks to be undefined for the next test).

Given the low per-file rate, I'd flag this as a worker-isolation issue to dig into (likely a shared mock or setup file) rather than something to fix per test file.

Root cause found in app/core/SDKConnect/SDKDeeplinkProtocol/DeeplinkProtocolService.ts:

public constructor() {
    if (!this.isInitialized) {
      this.init()
        .then(() => {
          this.isInitialized = true;
          DevLogger.log('DeeplinkProtocolService:: initialized');
        })
        .catch((err) => {
          this.isInitialized = false;
          Logger.log(err, 'DeeplinkProtocolService:: error initializing');
        });
    }
}

The constructor fires an un-awaited init() promise chain as a side effect. DeeplinkProtocolService.test.ts's beforeEach calls service = new DeeplinkProtocolService(); 14+ times, and a constructor return value can't be awaited — so every dangling .then()/.catch() callback (referencing DevLogger.log/Logger.log, both jest.mock()'d) resolves on a later microtask/macrotask. When the last of these fires after this test file's suite is done and Jest has already torn down its module environment (and possibly moved the worker to the next test file), it throws exactly the observed ReferenceError: You are trying to import a file after the Jest environment has been torn down, and can corrupt shared worker state for whatever unrelated test runs next in that worker — matching the cascading TypeError: Cannot read properties of undefined (reading 'some') / selectAccountByScope is not a function failures across totally unrelated files.

This is category J1 (missing await on async side effects) from the skill.

Proposed fix: add an afterEach in DeeplinkProtocolService.test.ts that flushes pending promises (using the existing flushPromises helper from app/util/test/utils.js) so the constructor's dangling init() chain always settles before the test/file ends, instead of leaking into the next test file's environment.

---

Note

created https://consensyssoftware.atlassian.net/browse/MCWP-699 for the above issue that the skill found.

Additional Context

Includes references/gh-analysis.md (GitHub Actions audit procedure) and repos/metamask-mobile.md (repo-specific flakiness pattern catalog, J1–J10).

fixes https://consensyssoftware.atlassian.net/browse/MCWP-473

[Copilot]

Pull request overview

Adds a new flaky-test-detection skill intended to help MetaMask Mobile contributors identify, prevent, and remediate flaky Jest unit tests, with both PR/diff review guidance and a GitHub Actions run-history audit procedure.

Changes:

• Introduces the flaky-test-detection skill entrypoint with review vs audit mode guidance.
• Adds a MetaMask Mobile–specific flakiness pattern catalog (J1–J10) with example fixes and a local reproduction loop.
• Documents an audit procedure for mining GitHub Actions logs to rank historically flaky test files by failure rate.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
domains/coding/skills/flaky-test-detection/skill.md	Adds the top-level skill description, constraints, and mode selection guidance.
domains/coding/skills/flaky-test-detection/repos/metamask-mobile.md	Adds the MetaMask Mobile pattern table (J1–J10) with fix guidance and reproduction commands.
domains/coding/skills/flaky-test-detection/references/gh-analysis.md	Adds the GitHub Actions audit procedure and ranking output format for flaky tests.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.