Repository navigation
fix: parse_url handles relative references like java.net.URI - #7392
Conversation
5e6ef04 to
1b6f970
Compare
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1b6f970 to
778984c
Compare
comphead
left a comment
There was a problem hiding this comment.
Thanks for the fix. I compared a scratch copy of decompose_uri and parse_url_component from this head against java.net.URI on JDK 17, over a hand list and about 60k random short strings. The relative-reference results agree for every part, apart from host and character validity cases. I have not built Comet or run its tests.
The description notes that scheme-less inputs now reach the #7330 character gaps. That turns NULLs that matched Spark into wrong values. java.net.URI rejects /p|q, /p%zz, /p^q, /p"q, /p\q, /p[1], ?a=b|c and #a#b, so Spark returns NULL (or INVALID_URL under ANSI). main returned NULL for them, but this head returns, for example, PATH /p|q. |, " and a stray % are plausible in logged page paths, the #7385 case. How many of the ~40k real paths in your differential run fall in this group? Could the character checks for path, query, fragment and % escapes be tightened in this PR, or land first?
Not caused by this PR, and I did not see them in #7330: opaque URIs (mailto:x gives PATH x, Spark NULL) and http: or user_id:5 (Spark rejects them, Comet returns a PROTOCOL).
| // A value without a scheme is a relative reference; a leading ':' is an empty | ||
| // scheme name, which java.net.URI rejects ("Expected scheme name at index 0"). | ||
| if scheme.is_none() && value.starts_with(':') { | ||
| return Err(invalid_url_err(value)); | ||
| } | ||
|
|
||
| // java.net.URI rejects an empty authority ("//", "http://") unless a path, | ||
| // query or fragment follows it ("Expected authority"). | ||
| if authority == Some("") && path.is_empty() && query.is_none() && fragment.is_none() { | ||
| return Err(invalid_url_err(value)); | ||
| } |
There was a problem hiding this comment.
scheme.is_none() && is redundant, since decompose_uri only reports a scheme when the : is at an index above 0, so value.starts_with(':') is enough. In the second check, file is the path plus any ?query, so file.is_empty() && fragment.is_none() says the same thing. I read this from decompose_uri and have not run it.
There was a problem hiding this comment.
Done, both simplified as you suggested.
| (':foo'), | ||
| ('//'), | ||
| ('http://'), | ||
| ('http://example.com/r?to=https://y.com&id=9') |
There was a problem hiding this comment.
The other tables in this file include a (NULL) row, but this one has none. Could you add (NULL) and ('')? The empty string returns before decompose_uri (PATH and FILE give '', the rest NULL), so it is a separate branch right next to the new code.
There was a problem hiding this comment.
Added ('') and (NULL), and (':') next to (':foo').
| fn test_invalid_relative_reference() { | ||
| for url in [":", ":foo", "://missing-scheme", "//", "http://"] { | ||
| for part in ["HOST", "PATH", "PROTOCOL", "FILE"] { | ||
| assert!( | ||
| parse_url_component(url, part, None).is_err(), | ||
| "{url} {part}" | ||
| ); | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
://missing-scheme is already covered by test_malformed_with_scheme_separator and by the ANSI and parse_url_enabled.sql fixtures. :foo, // and http:// are in the new SQL table, and :foo and http:// are also in the ANSI file. Both new branches return before part is read, so the loop over four parts checks the same early return four times. Most rows of test_relative_reference repeat the SQL table too, which is compared with Spark directly. Could the Rust tests keep only what the SQL files do not cover?
There was a problem hiding this comment.
Removed both. The SQL files cover those cases and compare with Spark directly. I kept one small Rust test, test_invisible_chars, for characters that are hard to see in a SQL file (DEL, NBSP, NEL, U+2028).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for checking this against You're right about the character gaps. In my run, 340 of the ~40k real page paths (0.85%) were invalid for I tightened the checks in this PR (81e5b3b):
I also fixed the two cases you found outside this PR, since they sit in the same function:
The character checks apply to absolute URLs too, so they also cover the With this commit, all of the ~40k real paths, with and without a host prepended, match Spark 4.1.1. On the ~110k generated URLs, no row that matched before stops matching, and the rows that match Spark go from 61k to 96k (out of 110k). The rest differ in the authority (registry-based hosts and IPv6 literals), which is #7330. The new cases are in the |
andygrove
left a comment
There was a problem hiding this comment.
The PR backlog is huge and the maintainers don't have the bandwidth to review them all, so here is an automated review. Feel free to ignore it.
I found no blocking concerns. The change to parse_url_component in native/spark-expr/src/url_funcs/parse_url.rs matches java.net.URI for relative references, opaque URIs, scheme names and the character checks, which is what Spark's ParseUrl relies on.
To check this I copied decompose_uri, has_invalid_uri_chars and parse_url_component from this head into a scratch crate. I compared all eight parts (PROTOCOL, HOST, PATH, QUERY, REF, FILE, AUTHORITY, USERINFO) against new URI(...) on JDK 17 for 300k random strings built from tricky tokens such as :, //, %zz, [, |, NBSP, NEL, U+2028 and U+180E. Every difference was in the authority, either HOST of a registry-based authority, an unbalanced bracket pair, or USERINFO of an @ authority. That is the remaining part of #7330 and the PR already says so. The non-ASCII rule (control or whitespace rejected, everything else allowed) agreed with Character.isISOControl and Character.isSpaceChar on all of those inputs.
Against branch-1.2 the behavior changes are all intended and all move Comet toward Spark. Scheme-less values used to return NULL for every part and now return the parsed parts. http://, http:, user_id:5 and 1http://x used to return a PROTOCOL and now give INVALID_URL under ANSI or NULL otherwise. /r?to=http://x.com used to raise INVALID_URL and no longer does. Opaque URIs such as mailto:x used to return PATH x and now return only PROTOCOL and REF. I did not find an input where an answer that matched Spark on the release now differs.
The new has_invalid_uri_chars runs on every row, so I timed it against the old version on 100k URLs of about 110 bytes each. Both took roughly 80 ns per URL, so the stricter validation does not add measurable cost.
The SQL tables in parse_url.sql and parse_url_ansi.sql compare against Spark and cover the relative, character and scheme cases, and test_invisible_chars covers the characters a SQL file cannot show. I have no further requests.
comphead
left a comment
There was a problem hiding this comment.
Thanks for addressing the earlier points. In 81e5b3b2ca the scheme-less |, " and %zz paths are rejected again, the redundant scheme.is_none() and is_empty checks are gone, the SQL tables have (NULL) and ('') rows, and the Rust tests are down to test_invisible_chars. I copied the new parsing functions into a scratch program and compared them with new URI(...) on JDK 11 over about 900k random strings. PROTOCOL, PATH, QUERY, REF and FILE agree wherever both accept. The remaining differences are all in the authority: Comet accepts some strings with unbalanced or misplaced IPv6 brackets (for example //h[::1]x and //&&[]) that java.net.URI rejects, and HOST or USERINFO differ for registry-based authorities. That is the part of #7330 this PR leaves open. I have not built Comet or run its tests.
Which issue does this PR close?
Closes #7385.
Rationale for this change
Native
parse_url/try_parse_urlreturn NULL for every part of a URL without a scheme (/p?id=1,?id=7,//host/p,notaurl). Spark'sParseUrlusesjava.net.URI, which parses these as relative references, so with Comet enabledparse_url(page, 'QUERY', 'gclid')on logged page paths silently returns NULL instead of the value.What changes are included in this PR?
In
parse_url_component, the earlyreturn Ok(None)for scheme-less values is replaced by the two cases wherejava.net.URIactually rejects such a value::(empty scheme name, "Expected scheme name at index 0") →INVALID_URL;//,http://, "Expected authority") →INVALID_URL.Everything else goes through the existing
decompose_uri/ part extraction, which already splits relative references per RFC 3986 Appendix B.Side effects, both matching Spark:
://inside a relative URL's query (/r?to=http://x.com&id=1) no longer raisesINVALID_URL.http://(scheme, empty authority, nothing else) used to return PROTOCOLhttp; it now raisesINVALID_URL(ANSI) / returns NULL, as Spark does.Not in scope: Comet's character validation (
has_invalid_uri_chars) is narrower thanjava.net.URI's (e.g.|, a malformed%escape, a second#), andextract_hostdoesn't model registry-based authorities (//ex_ample.com/p). Those gaps already exist for absolute URLs and are tracked in #7330; with this change scheme-less inputs reach them too (Spark returns NULL for such inputs, Comet now returns parsed values instead of NULL).How are these changes tested?
parse_url.sql, plus relative and error cases inparse_url_ansi.sql.CometSqlFileTestSuite parse_urlpasses locally (Spark 4.1 profile) and fails onmainat the new queries.parse_url.rsfor the changed branch:test_relative_reference(a few relative references per part, expectations from Spark 4.1.1) andtest_invalid_relative_reference(:,:foo,://missing-scheme,//,http://). Both fail onmain; all existingspark-exprtests pass,cargo fmtandclippy -D warningsare clean.try_parse_url+java.net.URIvalidity on ~110k generated URLs and ~40k real-world relative page paths (each also tried with a host prepended): no change in results for absolute URLs except thehttp://-style inputs above; relative references now match Spark except the inputs covered by Native parse_url accepts URLs and registry-based hosts that java.net.URI treats differently #7330.cargo bench --bench parse_url(main vs this branch): existing absolute-URL cases unchanged (all within ±4%). Scheme-less rows now do the normal extraction instead of returning early.🤖 Generated with Claude Code
UPD: after review
java.net.URIfor relative and absolute URLs:",\,^,`,|, DEL and non-ASCII control or space characters, a%not followed by two hex digits,[or]in a hierarchical path, and a second#giveINVALID_URL(ANSI) or NULL. This covers the character rows of Native parse_url accepts URLs and registry-based hosts that java.net.URI treats differently #7330, so the "Not in scope" paragraph above no longer applies to them.user_id:5,1http://…are invalid), and a scheme with nothing after the:(http:,http:#f) is invalid.mailto:x,urn:isbn:123,http:p?id=1) return only PROTOCOL and REF, as Spark does.extract_host(//ex_ample.com/p, the host rows of Native parse_url accepts URLs and registry-based hosts that java.net.URI treats differently #7330).test_parse_url_charsandtest_parse_url_schemestables inparse_url.sqland error cases inparse_url_ansi.sql.test_relative_referenceandtest_invalid_relative_referencewere replaced bytest_invisible_chars, since the SQL files cover the rest. The differential run was repeated. All ~40k real relative paths, with and without a host prepended, match Spark 4.1.1. On the ~110k generated URLs, no row that matched before stops matching, and matching rows go from 61k to 96k. The remaining differences are in the authority, which is Native parse_url accepts URLs and registry-based hosts that java.net.URI treats differently #7330.