fix: scan whole URL past sub-delims in exfil redactor (#7611) - #7666
fix: scan whole URL past sub-delims in exfil redactor (#7611)#7666patrigao wants to merge 1 commit into
Conversation
Design Review (Fable 5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS Sound root-cause fix on the backend, but the same emission-grammar bug it names stays live in the frontend mirror scanner. WatchThe PR diagnoses the root cause as an emission grammar leaked into a scanner, yet Suggestions
[DESIGN-REVIEWED] d8194dd |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of All facts are verified. The strongest finding: First-Principles-Verdict: CONCERNS #7611's own scheduled subtraction is skipped: Not justified as shipped
What this change shipsIntent: stop a legal
Watch
Subtractions
[FIRST-PRINCIPLES-REVIEWED] d8194dd |
Opus 4.8 Review — ✅ no blocking findingsReviewed Review detailsI've independently traced the changed code in
Nothing survives to the 80+ bar; the discovery pass's "no candidates" holds under independent falsification, and I found no groundable Step-2 addition. No findings. [OPUS-REVIEWED] d8194dd Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
d3be11b to
7733651
Compare
|
🤖 Kiro Crew Auto-Pipeline [operator: patrigao]: Addressed both red gates on head 7733651.
|
GPT 5.6 Review — 🔴 changes requested (blocking)GPT 5.6 found at least one blocking issue that must be resolved before merging This comment is updated in place on each push. BLOCKING -- src/kiro_crew/security/exfil.py:115 -- URL scanner still truncates browser-delivered hrefs Adjudication (Opus 4.8) — is blocking on each finding proportionate?I cannot execute code in this environment, so I ruled on the regex semantics directly, which are decisive here. F1 analysis (fenced). The tail char class uses the negative lookahead
This is the exact truncation-bypass class the PR set out to close, on the credential-egress redaction ceiling. Harm rung: UNBOUNDED (secret/credential exfiltration past the redaction gate). Recovery path: none — once the unredacted link renders and is clicked, the secret is delivered irreversibly. Rarity: the trigger is model-authored/injection-steered text carrying a crafted URL — precisely this module's stated threat model (exfil.py:1106-1123), not an extreme or self-contradicting condition an operator's own writer cannot produce. I cannot complete a rarity record that would justify a FLAG. Adjudicable (non-fenced) block is empty. False positive or not applicable? A repository writer can comment: |
7733651 to
153efd3
Compare
Confirmed legitimate. Fixed with a stronger invariant than the suggested "split only after a Regression tests: |
Confirmed legitimate — the Regression test: |
153efd3 to
63afb4d
Compare
Confirmed legitimate; applied the suggested fix. The trailing- Span note for the ledger: this is the second round landing in |
Legitimacy check (done before escalating): the chain is reachable — Why I am not pushing a fourth patch: this is the third consecutive round landing in the same span, and all three findings relitigate one design question — which delivery-channel model defines where a URL span ends?
The class currently stops at whitespace + The ruling requested — pick one:
Option 1 is my recommendation: it is the invariant that makes all three rounds' findings unreachable at once, rather than a fourth point-fix. |
Open PR relationship auditThis is a consolidated, point-in-time code-level audit note. It compares complete merge-base diffs and current/merged code; it does not treat a shared topic as duplication or partial coverage as completion. Relationship findings
No PR, Issue, label, branch, or review state was changed by the relationship-note portion of this audit. |
63afb4d to
e347365
Compare
|
Rebased onto main Conflicts and how they were resolved:
Gates run locally on changed files: Please review the resolution. A maintainer push makes the maintainer the last pusher, so a second approver is needed under the repo's last-push rule. Reply if anything looks wrong. |
|
Reviewed the resolution — it is faithful. I diffed the added source lines of the pre-rebase head ( Doc and test deltas vs main match the original PR exactly ( Note for reviewers: the round-5 needs-a-decision escalation (comment 5501586422) is still open — the |
e347365 to
1a241e2
Compare
1a241e2 to
d8194dd
Compare
|
Update on the stalled The PR was rebased twice today (maintainer's open-PR-audit rebase onto
This is the same class the escalation is about (host-boundary lookahead is structurally fragile), and it argues for option 1 (wrapper-pair model) over a point-fix: option 1 removes the host-boundary lookahead entirely, so Still holding for a ruling — one of: (1) implement the wrapper-pair model; (2) |
Problem / Motivation
The shared exfiltration-URL redactor's
_URL_RE(src/kiro_crew/security.py) excluded)and'from its path/query character class. A URL carrying either character in its path truncated the match before the?, so_exfil_url_warningreceived apath_and_querywith no query and returned clean at itsqmark == -1early return. Everything after the truncation — the entire query string — escaped every scan that follows (hard-credential pass, fixed-signature pass, percent-decode loop, and the base64/entropy heuristic).Verified repro through
_redact_provider_data:https://evil.example.com/a?data=<40-char-blob>is redacted;https://evil.example.com/a)b?data=<same-blob>passed clean. The reporter's measured follow-up showed the whole terminating class was affected (),',",>, whitespace), and an apostrophe in a page title is enough to trip it — no encoding tricks needed.Why it matters
redact_provider_data/redact_exfiltration_urlsrun on every provider- or model-controlled string that reaches an output surface (issue/PR bodies, review comments, Jira strings). A single ordinary character in a URL let a high-entropy query blob skip the entropy and credential checks — a real data-exfiltration bypass. This is the third truncation bypass of the same class after the raw-IP-host and no-path-query fixes the regex comment already documents.What changed (motivation → approach → change)
Root cause: the path/query character class inherited an emission grammar (which characters end a URL when it is put into a markdown destination or a quoted string) and applied it to the scanner (which wants the whole URL). Per the reporter's suggested direction — permissive for classification, strict only for emission:
",>,`(all illegal unencoded in a URL per RFC 3986, so every linkifier/renderer/unfurler stops there).)and'are RFC 3986 sub-delims — legal URL content — and are now scanned through._url_scan_span(match)performs a wrapper-aware trim before classification and redaction: a trailing wrapper quote or)is stripped ONLY on structural proof of a wrapper — the same quote, or the matching(, must immediately PRECEDE the URL — and never more than one byte (round 3: the earlier while-loop trimmed every unbalanced trailing), hiding a pure)-run from the length heuristic; round 4: without the preceding-(proof, a bare URL's legal trailing)byte was dropped and an exactly-threshold query slipped under the length check). It never strips a run and never truncates at an interior'/)(which would reopen the bypass for a quote-wrapped path that legitimately contains an apostrophe). The residual is a bounded false positive (over-redacting a quoted URL immediately followed by a quoted field, or over-scanning the stray closer of a double-wrapped((url))) — the safe direction.(?!https?://<host>), whose host alternation is shared with the captured host group via one_URL_HOSTconstant, splits back-to-back URLs so a glued second URL is classified under its own host. It requires a real host after the inner scheme, so it fires only where the split yields a scanned match — a bare(?!https?://)split onhttps://evil/https://?key=<secret>(inner scheme, no host) produced no second match and left the first match'sgroup(3)as just/, reopening theqmark == -1bypass.https://evil.tld/?q=)https://tenant.tld/?nav=<base64>). Both scan and redact now deny the exact-host and presigned exemptions for glued spans and run the full heuristics; standalone exempt URLs are unchanged. This is deliberately stronger than narrowing the split to wrapper-adjacent positions, since the glue character is attacker-chosen.re.IGNORECASE | re.ASCIIon the pattern: the scheme is case-insensitive per RFC 3986 §3.1 (an uppercaseHTTPS://a browser follows must not bypass the scan), andre.ASCIIkeeps the fold ASCII-only so a Unicode case-fold (s↔ſ,k↔K) cannot widen the scheme/TLD classes.redact_exfiltration_urlsnow splices each classified match by span, right-to-left, instead of a globalstr.replace(url, …)that could rewrite a longer URL sharing the same leading bytes and leave its tail behind.Minimal blast radius: scan-side widening only. Markdown emission (
#7543's_md_link_target, which is not in this tree) is untouched.Tests
New cases in
test/test_security.py(TestExfilUrlPathAndRawIp) andtest/test_source_providers.py:_redact_provider_data(the named entry point);?) is still scanned;"/>/backtick still terminate, with content before them still scanned; a compact-JSON sha adjacent to a"-wrapped URL stays clean;)on redaction; a markdown-wrapped S3 presigned URL keeps its exemption; a benign balancedFoo_(bar)wiki URL stays clean;)'-run query is not stripped wholesale; a PURE 250×)trailing run stays in the scanned span (round 3); a glued exempt-tenant span loses its exemption while a whitespace-separated one keeps it (round 3); a bare URL's trailing)byte counts toward the length threshold while a wrapped URL still sheds only its wrapper (round 4); the inner-scheme-no-host case is still scanned; the trailing-paren trim is linear (100k-)run returns promptly); an uppercase and a Unicode-fold scheme do not act as a boundary; span-precise redaction of prefix-related URLs.Manual verification
N/A — unit coverage is sufficient; this is a pure text-classification change exercised end-to-end through
scan_exfiltration_urls,redact_exfiltration_urls, and_redact_provider_data.Related Issues
Fixes #7611
Pattern harvest
Rule candidate: review-prompt
Pattern: an emission/serialization grammar (which characters terminate a token when it is placed into markdown/HTML/a quoted string) reused as a scanner grammar — the scanner truncates on a legal-content character and everything past it escapes inspection. Three instances now in
_URL_REalone (raw-IP host, no-path query, sub-delim path). When a validator and an emitter share a character class, check whether the validator wants the maximal token and the emitter the delimited one.