diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d4a39380..3c414ffd 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -425,18 +425,26 @@ that omits them passes where CI fails; `tests/e2e-coverage-gate.test.mjs` holds floors under them. Invalid maps, stale manifests, and empty aggregate attribution fail visibly and leave their raw artifacts for review. -Read the percentages against the Node major CI pins. These are _region_ counts, -and V8 splits the same expression into a different number of ranges on a -different runtime, so the denominator moves even though the code, the suite and -the fixture builds do not. The same commit measures 97.29% source regions on the -major the job pins and 97.73% on a newer runtime, and per file the gap is wider -still: `DirectoryFreshness.js` is 11/15 (73.33%) on the pinned major (node -22.23.3) and 9/10 (90.00%) on v26.10.0. A local render on a newer runtime -therefore reports headroom the gate does not have, and ratcheting a floor to -match it turns the required job red. `tests/e2e-coverage-gate.test.mjs` records -the major its floors were measured on and fails if the job moves off it. The -counts also move between patch releases of that major, so a small same-major -difference is runtime drift, not a code change. +Compare percentages from equivalent build, browser and reporting toolchains, +including the Node major CI pins. These are _region_ counts: the denominator can +move between toolchains even when the code and suite do not. The same commit +measures 97.29% source regions on the major the job pins and 97.73% on a newer +runtime, and per file the gap is wider still: `DirectoryFreshness.js` is 11/15 +(73.33%) on the pinned major (node 22.23.3) and 9/10 (90.00%) on v26.10.0. A +local render on a newer runtime therefore reports headroom the gate does not +have, and ratcheting a floor to match it turns the required job red. +`tests/e2e-coverage-gate.test.mjs` records the major its floors were measured on +and fails if the job moves off it. The counts also move between patch releases +of that major, so a small same-major difference is not by itself evidence of a +code change. + +Reports distinguish `buildRuntimes`, `captureRuntimes` and `reporterRuntime`. +Coverage builds record their Node/V8 identity; captures record Node/V8, +Playwright and the browser's product/V8 versions. Chromium V8 produces the raw +browser ranges, not the reporting process's V8. Sealing preserves producer +identities for later re-renders, while `reporterRuntime` identifies the current +renderer. Legacy artifacts explicitly report unknown producer metadata rather +than attributing it to the renderer. Some component branches render only for data shapes the checked-in `data/*.json` never take — an archived End User Group, a person with neither role nor company, diff --git a/adr/0007-scope-end-to-end-coverage-to-user-reachable-states.md b/adr/0007-scope-end-to-end-coverage-to-user-reachable-states.md index aef957a8..03cbb8b5 100644 --- a/adr/0007-scope-end-to-end-coverage-to-user-reachable-states.md +++ b/adr/0007-scope-end-to-end-coverage-to-user-reachable-states.md @@ -95,6 +95,33 @@ restrictions on test-tree guards. It does not authorize data-shape browser fixture builds, change coverage floors, or replace surviving-mutant evidence for assertion gaps within mutation-covered production logic. +## Clarification: measurement attribution and provenance (2026-10-21) + +The calibration in [#1752](https://github.com/cncf/endusers/issues/1752) and +provenance repair in [#1748](https://github.com/cncf/endusers/pull/1748) +motivate these evidence requirements. + +A mutation result of `NoCoverage` means the configured run did not attribute +execution to that mutant; it does not by itself prove an assertion gap. +Calibrate the runner and any subprocess boundary, including active-mutant +selection and child-to-parent coverage reporting, before treating those results +as missing assertions or using them to justify mutation-scope or threshold +changes. Assertion gaps within mutation-covered production logic still require +the surviving-mutant evidence in decision 3. + +Browser-coverage artifacts preserve producer identities separately from the +current reporter runtime: build-process Node/V8 identity, and capture-process +Node/V8, Playwright and browser product/V8 identities. Record producer metadata +when building or capturing and preserve it when sealing the run. Later renders, +including failure reports, retain those identities and identify the current +reporter separately. Missing legacy producer metadata remains explicitly +unknown, not inferred from the renderer. The reporter's runtime alone is not +evidence of which browser produced the raw ranges or why region counts differ. + +This clarification selects no new mutation-runner architecture, expands no +mutation scope, changes no thresholds, and does not relax the existing +browser-fixture or test-tree restrictions. + ## Consequences - Hive `quality` findings that do not meet decision 2 are closed as not planned diff --git a/docusaurus.config.js b/docusaurus.config.js index 35af5747..21a533c2 100644 --- a/docusaurus.config.js +++ b/docusaurus.config.js @@ -54,6 +54,14 @@ const IMAGE_HOST_SOURCES = [ function endusersE2ESourceMaps() { return { name: 'endusers-e2e-source-maps', + async postBuild({ outDir }) { + const { writeFile } = require('node:fs/promises'); + const { join } = require('node:path'); + await writeFile( + join(outDir, 'e2e-coverage-runtime.json'), + JSON.stringify({ node: process.version, v8: process.versions.v8 }), + ); + }, configureWebpack(config, isServer) { if (isServer) return {}; return { devtool: 'source-map' }; diff --git a/tests/e2e-coverage-fixture.test.mjs b/tests/e2e-coverage-fixture.test.mjs index 8fa3d30d..3d8c7509 100644 --- a/tests/e2e-coverage-fixture.test.mjs +++ b/tests/e2e-coverage-fixture.test.mjs @@ -31,9 +31,21 @@ function info(testId = 'fixture-test') { }; } -function fakePage({ result = [], startError, stopError } = {}) { +function fakePage({ result = [], startError, stopError, versionError } = {}) { let started = false; return { + context: () => ({ + async newCDPSession() { + return { + async send(method) { + assert.equal(method, 'Browser.getVersion'); + if (versionError) throw versionError; + return { product: 'Chrome/123.0', jsVersion: '12.3' }; + }, + async detach() {}, + }; + }, + }), coverage: { async startJSCoverage() { if (startError) throw startError; @@ -111,6 +123,13 @@ test('CoverageManager writes a raw artifact with source identity metadata', asyn const payload = JSON.parse(await readFile(join(runDir, artifact), 'utf8')); assert.equal(payload.result[0].sourceLength, 3); assert.match(payload.result[0].sourceSha256, /^[a-f0-9]{64}$/); + assert.equal(payload.runtime.browser, 'Chrome/123.0'); + assert.equal(payload.runtime.browserV8, '12.3'); + assert.equal(payload.runtime.node, process.version); + assert.equal( + payload.runtime.playwright, + require('@playwright/test/package.json').version, + ); } finally { await rm(runDir, { recursive: true, force: true }); } @@ -184,15 +203,7 @@ test('CoverageManager writes honest error artifacts for start and stop failures' try { await initCoverageRun(runDir, 'errors-1'); const manager = new CoverageManager(info(), runDir, 'errors-1'); - await assert.rejects(() => - manager.startPage({ - coverage: { - async startJSCoverage() { - throw {}; - }, - }, - }), - ); + await assert.rejects(() => manager.startPage(fakePage({ startError: {} }))); await assert.rejects( () => manager.startPage({ coverage: {} }), (error) => { @@ -215,6 +226,26 @@ test('CoverageManager writes honest error artifacts for start and stop failures' } }); +test('CoverageManager records browser-version failures as capture errors', async (t) => { + const runDir = await mkdtemp(join(tmpdir(), 'endusers-fixture-version-')); + t.after(() => rm(runDir, { recursive: true, force: true })); + await initCoverageRun(runDir, 'version-error'); + const manager = new CoverageManager(info(), runDir, 'version-error'); + await assert.rejects( + () => + manager.startPage( + fakePage({ versionError: new Error('version failed') }), + ), + /version failed/, + ); + const artifact = (await readdir(runDir)).find((name) => + name.endsWith('-error.json'), + ); + const payload = JSON.parse(await readFile(join(runDir, artifact), 'utf8')); + assert.equal(payload.runtime, null); + assert.equal(payload.error.message, 'version failed'); +}); + test('CoverageManager aggregates manual-context cleanup errors', async () => { const runDir = await mkdtemp( join(tmpdir(), 'endusers-fixture-context-errors-'), diff --git a/tests/e2e-coverage-report.test.mjs b/tests/e2e-coverage-report.test.mjs index c54a4e10..f2019f75 100644 --- a/tests/e2e-coverage-report.test.mjs +++ b/tests/e2e-coverage-report.test.mjs @@ -82,11 +82,13 @@ async function writePageCoverage( pageIndex, ranges, url = 'http://localhost:3000/assets/js/app.js', + runtime, ) { await writeCoverageArtifact(fixture.runDir, `worker-0-page-${pageIndex}`, { schemaVersion: 1, kind: 'endusers.playwright.v8-coverage', runId: 'run-1', + runtime, result: [ { url, @@ -204,6 +206,13 @@ test('main surfaces recorded capture-error artifacts instead of treating them as test('collectE2ECoverage maps V8 ranges through an external source map to src lines', async () => { const fixture = await fixtureRun(); + const runtime = { + node: 'v20', + v8: '11', + playwright: '1.64', + browser: 'Chrome/123', + browserV8: '12.3', + }; try { const source = 'const hit = 1;\nconst miss = 2;\n'; const generated = source; @@ -230,14 +239,20 @@ test('collectE2ECoverage maps V8 ranges through an external source map to src li ]), }), ); - await writePageCoverage(fixture, 0, [ - { - startOffset: generated.indexOf('const miss'), - endOffset: scriptText.length, - count: 0, - }, - { startOffset: 0, endOffset: scriptText.length, count: 1 }, - ]); + await writePageCoverage( + fixture, + 0, + [ + { + startOffset: generated.indexOf('const miss'), + endOffset: scriptText.length, + count: 0, + }, + { startOffset: 0, endOffset: scriptText.length, count: 1 }, + ], + undefined, + runtime, + ); await sealCoverageRun(fixture.runDir, 'passed'); const report = await collectE2ECoverage(fixture.runDir, { @@ -245,6 +260,13 @@ test('collectE2ECoverage maps V8 ranges through an external source map to src li buildDir: fixture.buildDir, }); assert.equal(report.status, 'ok'); + assert.deepEqual(report.captureRuntimes, [runtime]); + assert.equal(report.reporterRuntime.node, process.version); + const rerendered = await collectE2ECoverage(fixture.runDir, { + root: fixture.root, + buildDir: fixture.buildDir, + }); + assert.deepEqual(rerendered.captureRuntimes, [runtime]); assert.deepEqual(report.sources, [ { file: 'src/components/Example/index.js', diff --git a/tests/e2e-coverage-runtime.test.mjs b/tests/e2e-coverage-runtime.test.mjs new file mode 100644 index 00000000..d55a996e --- /dev/null +++ b/tests/e2e-coverage-runtime.test.mjs @@ -0,0 +1,212 @@ +// Captured browser ranges, built JavaScript and report conversion can come +// from different runtimes. Re-rendering must not relabel the original producers. + +import assert from 'node:assert/strict'; +import test from 'node:test'; +import { mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; + +import { main, renderE2ECoverageReport } from './tools/e2e-coverage-report.mjs'; +import { + initCoverageRun, + readCoverageRun, + sealCoverageRun, + writeCoverageArtifact, +} from './tools/e2e-coverage-run.mjs'; + +/** A minimal report shell, so the two render cases differ only in runtime. */ +function reportWith(reporterRuntime) { + return { + status: 'ok', + runId: 'run-1', + runStatus: 'passed', + revision: null, + reporterRuntime, + sources: [], + summary: { + executableLines: 1, + coveredLines: 1, + linePercent: 100, + regions: 1, + coveredRegions: 1, + regionPercent: 100, + }, + unmappedSources: [], + missingSourceFiles: [], + diagnostics: { warnings: [], errors: [] }, + }; +} + +test('the text report distinguishes build, capture and reporter runtimes', () => { + const report = reportWith({ node: 'v22.23.3', v8: '12.4.254.21-node.26' }); + report.buildRuntimes = [{ build: '.', runtime: { node: 'v20', v8: '11' } }]; + report.captureRuntimes = [ + { + node: 'v21', + v8: '12', + playwright: '1.64', + browser: 'Chrome/123', + browserV8: '12.3', + }, + ]; + const text = renderE2ECoverageReport(report); + assert.match( + text, + /^Reporter runtime: node v22\.23\.3 \(v8 12\.4\.254\.21-node\.26\)$/mu, + ); + assert.match(text, /^Build runtimes: \.: node v20 \(v8 11\)$/mu); + assert.match( + text, + /^Capture runtimes: node v21 \(v8 12\), Playwright 1\.64, browser Chrome\/123 \(v8 12\.3\)$/mu, + ); + assert.match( + text, + /^Revision: unknown$/mu, + 'the runtime is a line of its own beside the revision, not folded into it', + ); +}); + +test('a report with no runtime says so rather than printing nothing', () => { + // The same shape `revision` uses: a consumer always gets the line, so a + // missing value reads as missing instead of as a report that never had the + // field. + assert.match( + renderE2ECoverageReport(reportWith(undefined)), + /^Reporter runtime: node unknown \(v8 unknown\)$/mu, + ); + const report = reportWith(undefined); + assert.match(renderE2ECoverageReport(report), /^Build runtimes: unknown$/mu); + assert.match( + renderE2ECoverageReport(report), + /^Capture runtimes: unknown$/mu, + ); + report.captureRuntimes = [null]; + report.buildRuntimes = [{ build: '.', runtime: null }]; + assert.match( + renderE2ECoverageReport(report), + /Playwright unknown, browser unknown \(v8 unknown\)/u, + ); +}); + +test('the failure artifact stamps the runtime it ran on', async (t) => { + const root = await mkdtemp(join(tmpdir(), 'endusers-e2e-runtime-')); + t.after(() => rm(root, { recursive: true, force: true })); + const runDir = join(root, 'coverage'); + const jsonPath = join(root, 'report.json'); + await initCoverageRun(runDir, 'run-runtime', { project: 'chromium' }); + // Sealed with no coverage artifacts at all, so the reporter refuses to + // publish a vacuous 0% and takes the error envelope instead. That envelope + // is the only record a run which broke before reporting leaves behind, + // which is the case where "which runtime was this?" matters most. + await sealCoverageRun(runDir, 'passed'); + + await assert.rejects(() => main(['--input', runDir, '--json', jsonPath])); + + const report = JSON.parse(await readFile(jsonPath, 'utf8')); + assert.equal(report.kind, 'endusers.e2e.coverage-report-error'); + // Read from the live process rather than hard-coded: the point of the stamp + // is that it reports the runtime that actually produced the numbers, so a + // literal here would pass while the field lied. + assert.deepEqual(report.reporterRuntime, { + node: process.version, + v8: process.versions.v8, + }); +}); + +test('sealing and re-rendering preserve producer identity on failure', async (t) => { + const root = await mkdtemp(join(tmpdir(), 'endusers-e2e-producers-')); + t.after(() => rm(root, { recursive: true, force: true })); + const runDir = join(root, 'coverage'); + const buildDir = join(root, 'build'); + const variant = 'e2e-coverage-variant'; + const legacy = 'e2e-coverage-legacy'; + await mkdir(join(buildDir, variant), { recursive: true }); + await mkdir(join(buildDir, legacy)); + await mkdir(join(buildDir, legacy, 'assets/js'), { recursive: true }); + await mkdir(join(buildDir, 'e2e-coverage-fixtures')); + const buildRuntime = { node: 'v18', v8: '10' }; + await writeFile( + join(buildDir, 'e2e-coverage-runtime.json'), + JSON.stringify(buildRuntime), + ); + await writeFile( + join(buildDir, variant, 'e2e-coverage-runtime.json'), + JSON.stringify({ node: 'v20', v8: '11' }), + ); + const captureRuntime = { + node: 'v21', + v8: '12', + playwright: '1.64', + browser: 'Chrome/123', + browserV8: '12.3', + }; + await initCoverageRun(runDir, 'producers'); + for (const stem of ['first', 'second']) + await writeCoverageArtifact(runDir, stem, { + kind: 'endusers.playwright.v8-coverage-error', + runId: 'producers', + runtime: captureRuntime, + error: { message: 'capture failed' }, + }); + await writeCoverageArtifact(runDir, 'legacy', { + kind: 'endusers.playwright.v8-coverage', + runId: 'producers', + result: [], + }); + const sealed = await sealCoverageRun(runDir, 'failed', { buildDir }); + assert.deepEqual(sealed.captureRuntimes, [captureRuntime, null]); + assert.deepEqual(sealed.buildRuntimes, [ + { build: '.', runtime: buildRuntime }, + { build: legacy, runtime: null }, + { build: variant, runtime: { node: 'v20', v8: '11' } }, + ]); + await writeFile( + join(buildDir, 'e2e-coverage-runtime.json'), + JSON.stringify({ node: 'changed', v8: 'changed' }), + ); + for (const name of ['first', 'rerender']) { + const jsonPath = join(root, `${name}.json`); + await assert.rejects( + () => main(['--input', runDir, '--json', jsonPath]), + /sealed as failed/, + ); + const report = JSON.parse(await readFile(jsonPath, 'utf8')); + assert.deepEqual(report.buildRuntimes, sealed.buildRuntimes); + assert.deepEqual(report.captureRuntimes, sealed.captureRuntimes); + assert.equal(report.reporterRuntime.node, process.version); + } +}); + +test('invalid capture JSON keeps an explicit provenance warning for the reporter', async (t) => { + const runDir = await mkdtemp(join(tmpdir(), 'endusers-e2e-runtime-invalid-')); + t.after(() => rm(runDir, { recursive: true, force: true })); + await initCoverageRun(runDir, 'invalid'); + await writeFile(join(runDir, 'bad.json'), '{ invalid'); + await writeFile(join(runDir, 'ignored.json'), 'null'); + const sealed = await sealCoverageRun(runDir, 'passed', { + buildDir: join(runDir, 'absent'), + }); + assert.equal(sealed.provenanceWarnings.length, 1); + assert.match( + sealed.provenanceWarnings[0], + /Capture runtime unavailable for bad.json/, + ); + assert.deepEqual(sealed.captureRuntimes, []); + assert.deepEqual(sealed.buildRuntimes, []); +}); + +test('malformed build runtime metadata fails sealing explicitly', async (t) => { + const root = await mkdtemp(join(tmpdir(), 'endusers-e2e-build-runtime-')); + t.after(() => rm(root, { recursive: true, force: true })); + const runDir = join(root, 'coverage'); + const buildDir = join(root, 'build'); + await mkdir(buildDir); + await writeFile(join(buildDir, 'e2e-coverage-runtime.json'), '{ invalid'); + await initCoverageRun(runDir, 'bad-build'); + await assert.rejects( + () => sealCoverageRun(runDir, 'passed', { buildDir }), + SyntaxError, + ); + assert.equal((await readCoverageRun(runDir)).status, 'started'); +}); diff --git a/tests/site-config.test.mjs b/tests/site-config.test.mjs index 109e8b4b..78852b75 100644 --- a/tests/site-config.test.mjs +++ b/tests/site-config.test.mjs @@ -299,6 +299,18 @@ test('e2e coverage source maps are opt-in and client-only', async () => { devtool: 'source-map', }); assert.deepEqual(plugin.configureWebpack({}, true), {}); + const outDir = mkdtempSync(join(tmpdir(), 'endusers-build-runtime-')); + try { + await plugin.postBuild({ outDir }); + assert.deepEqual( + JSON.parse( + readFileSync(join(outDir, 'e2e-coverage-runtime.json'), 'utf8'), + ), + { node: process.version, v8: process.versions.v8 }, + ); + } finally { + rmSync(outDir, { recursive: true, force: true }); + } }); test('the e2e data fixtures are opt-in and scoped to the site data directory', async () => { diff --git a/tests/tools/e2e-coverage-report.mjs b/tests/tools/e2e-coverage-report.mjs index 589a8e30..c7aec8cc 100644 --- a/tests/tools/e2e-coverage-report.mjs +++ b/tests/tools/e2e-coverage-report.mjs @@ -735,6 +735,16 @@ async function convertScript(scriptCoverage, root, scriptRoots) { }; } +// Producer identity belongs to the sealed artifacts, not the process that +// happens to render them later. +function reportProvenance(run) { + return { + reporterRuntime: { node: process.version, v8: process.versions.v8 }, + buildRuntimes: run?.buildRuntimes ?? null, + captureRuntimes: run?.captureRuntimes ?? null, + }; +} + function emptyReport(run, status = 'ok') { return { schemaVersion: 2, @@ -746,6 +756,7 @@ function emptyReport(run, status = 'ok') { // tests/tools/e2e-coverage-run.mjs. Null when the run could not name its // commit, never absent, so a consumer always has the key to check. revision: run.revision ?? null, + ...reportProvenance(run), scripts: { captured: 0, converted: 0, ignored: 0 }, sources: [], summary: { @@ -758,7 +769,7 @@ function emptyReport(run, status = 'ok') { }, unmappedSources: [], missingSourceFiles: [], - diagnostics: { warnings: [], errors: [] }, + diagnostics: { warnings: run.provenanceWarnings ?? [], errors: [] }, }; } @@ -996,6 +1007,10 @@ export async function collectE2ECoverage( return report; } +function nodeRuntime(runtime) { + return `node ${runtime?.node ?? 'unknown'} (v8 ${runtime?.v8 ?? 'unknown'})`; +} + export function renderE2ECoverageReport(report) { const lines = [ `E2E coverage: ${report.status} (run ${report.runId}; status ${report.runStatus ?? 'unknown'})`, @@ -1004,6 +1019,9 @@ export function renderE2ECoverageReport(report) { // scanning the artifact for which commit these line numbers address should // not have to parse a parenthetical to find it. `Revision: ${report.revision ?? 'unknown'}`, + `Build runtimes: ${report.buildRuntimes?.map(({ build, runtime }) => `${build}: ${nodeRuntime(runtime)}`).join('; ') || 'unknown'}`, + `Capture runtimes: ${report.captureRuntimes?.map((runtime) => `${nodeRuntime(runtime)}, Playwright ${runtime?.playwright ?? 'unknown'}, browser ${runtime?.browser ?? 'unknown'} (v8 ${runtime?.browserV8 ?? 'unknown'})`).join('; ') || 'unknown'}`, + `Reporter runtime: ${nodeRuntime(report.reporterRuntime)}`, '', // The region column is what makes a --check-source-regions failure // diagnosable from the text artifact alone, mirroring the unit reporter. @@ -1078,6 +1096,7 @@ export async function main(argv = process.argv.slice(2)) { runId: error.runId ?? run?.runId ?? null, runStatus: error.runStatus ?? run?.status ?? null, revision: run?.revision ?? null, + ...reportProvenance(run), scripts: { captured: 0, converted: 0, ignored: 0 }, sources: [], summary: { @@ -1096,7 +1115,10 @@ export async function main(argv = process.argv.slice(2)) { // reader gets of why the gate failed, and publishing `null` there would // be worse than publishing the coerced text. // tests/e2e-coverage-error-arms.test.mjs pins the premise. - diagnostics: { warnings: [], errors: [error.message ?? String(error)] }, + diagnostics: { + warnings: run?.provenanceWarnings ?? [], + errors: [error.message ?? String(error)], + }, }; await writeE2ECoverageReport(failure, { jsonPath: options.json, diff --git a/tests/tools/e2e-coverage-run.mjs b/tests/tools/e2e-coverage-run.mjs index c84d5f4b..5c2be099 100644 --- a/tests/tools/e2e-coverage-run.mjs +++ b/tests/tools/e2e-coverage-run.mjs @@ -145,6 +145,64 @@ export async function readCoverageRun(runDir) { return validateManifest(JSON.parse(raw)); } +async function readProvenance(runDir, runId, buildDir) { + const captures = new Map(); + const warnings = []; + for (const entry of await readdir(runDir)) { + if (!entry.endsWith('.json') || entry === 'manifest.json') continue; + const raw = await readFile(join(runDir, entry), 'utf8'); + let payload; + try { + payload = JSON.parse(raw); + } catch (error) { + if (!(error instanceof SyntaxError)) throw error; + warnings.push( + `Capture runtime unavailable for ${entry}: ${error.message}`, + ); + continue; + } + if ( + payload?.runId === runId && + [ + COVERAGE_ARTIFACT_KIND, + 'endusers.playwright.v8-coverage-error', + ].includes(payload.kind) + ) { + const runtime = payload.runtime ?? null; + captures.set(JSON.stringify(runtime), runtime); + } + } + const builds = []; + if (buildDir && (await exists(buildDir))) { + const variants = (await readdir(buildDir, { withFileTypes: true })) + .filter( + (entry) => + entry.isDirectory() && entry.name.startsWith('e2e-coverage-'), + ) + .map((entry) => entry.name) + .sort(); + for (const build of ['.', ...variants]) { + const path = join(buildDir, build, 'e2e-coverage-runtime.json'); + const hasRuntime = await exists(path); + if ( + build !== '.' && + !hasRuntime && + !(await exists(join(buildDir, build, 'assets/js'))) + ) + continue; + builds.push({ + build, + runtime: hasRuntime ? JSON.parse(await readFile(path, 'utf8')) : null, + }); + } + } + return { + captureRuntimes: Array.from(captures.values()), + buildRuntimes: builds, + provenanceWarnings: warnings, + }; +} + export async function sealCoverageRun( runDir, status, @@ -171,6 +229,7 @@ export async function sealCoverageRun( : null; const sealed = { ...manifest, + ...(await readProvenance(runDir, manifest.runId, buildDir)), status, sealedAt: new Date().toISOString(), ...(scripts diff --git a/tests/tools/e2e-coverage.cjs b/tests/tools/e2e-coverage.cjs index b9c27bf6..402ee0fa 100644 --- a/tests/tools/e2e-coverage.cjs +++ b/tests/tools/e2e-coverage.cjs @@ -1,6 +1,7 @@ const { createHash } = require('node:crypto'); const { test: baseTest, expect } = require('@playwright/test'); +const { version: playwrightVersion } = require('@playwright/test/package.json'); const managers = new Map(); const runModulePromise = import('./e2e-coverage-run.mjs'); @@ -71,6 +72,7 @@ class CoverageManager { parallelIndex: this.testInfo.parallelIndex, retry: this.testInfo.retry, pageIndex: pageInfo.pageIndex, + runtime: pageInfo.runtime ?? null, error: errorDetails(error), }); } @@ -85,24 +87,33 @@ class CoverageManager { pageIndex: this.pageIndex, }; this.pageIndex += 1; - if (empty) { - const record = { info, page, empty: true }; - this.pages.set(page, record); - return record; - } try { - if (typeof page.coverage?.startJSCoverage !== 'function') { + if (!empty && typeof page.coverage?.startJSCoverage !== 'function') { throw new Error('Playwright JavaScript coverage is unavailable'); } - await page.coverage.startJSCoverage({ - reportAnonymousScripts: false, - resetOnNavigation: false, - }); + const session = await page.context().newCDPSession(page); + try { + const version = await session.send('Browser.getVersion'); + info.runtime = { + node: process.version, + v8: process.versions.v8, + playwright: playwrightVersion, + browser: version.product, + browserV8: version.jsVersion, + }; + } finally { + await session.detach(); + } + if (!empty) + await page.coverage.startJSCoverage({ + reportAnonymousScripts: false, + resetOnNavigation: false, + }); } catch (error) { await this.writeError(info, error); throw error; } - const record = { info, page }; + const record = { info, page, empty }; this.pages.set(page, record); return record; } @@ -135,6 +146,7 @@ class CoverageManager { parallelIndex: this.testInfo.parallelIndex, retry: this.testInfo.retry, pageIndex: record.info.pageIndex, + runtime: record.info.runtime, result: normalized, }, );