Skip to content

fix: parse_url handles relative references like java.net.URI - #7392

Merged
andygrove merged 7 commits into
apache:mainfrom
valentinsoe:fix/parse-url-relative-urls
Nov 18, 2026
Merged

andygrove merged 7 commits into
apache:mainfrom
valentinsoe:fix/parse-url-relative-urls

Conversation

@valentinsoe

@valentinsoe valentinsoe commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #7385.

Rationale for this change

Native parse_url / try_parse_url return NULL for every part of a URL without a scheme (/p?id=1, ?id=7, //host/p, notaurl). Spark's ParseUrl uses java.net.URI, which parses these as relative references, so with Comet enabled parse_url(page, 'QUERY', 'gclid') on logged page paths silently returns NULL instead of the value.

What changes are included in this PR?

In parse_url_component, the early return Ok(None) for scheme-less values is replaced by the two cases where java.net.URI actually rejects such a value:

  • a leading : (empty scheme name, "Expected scheme name at index 0") → INVALID_URL;
  • an empty authority with nothing after it (//, http://, "Expected authority") → INVALID_URL.

Everything else goes through the existing decompose_uri / part extraction, which already splits relative references per RFC 3986 Appendix B.

Side effects, both matching Spark:

  • :// inside a relative URL's query (/r?to=http://x.com&id=1) no longer raises INVALID_URL.
  • http:// (scheme, empty authority, nothing else) used to return PROTOCOL http; it now raises INVALID_URL (ANSI) / returns NULL, as Spark does.

Not in scope: Comet's character validation (has_invalid_uri_chars) is narrower than java.net.URI's (e.g. |, a malformed % escape, a second #), and extract_host doesn't model registry-based authorities (//ex_ample.com/p). Those gaps already exist for absolute URLs and are tracked in #7330; with this change scheme-less inputs reach them too (Spark returns NULL for such inputs, Comet now returns parsed values instead of NULL).

How are these changes tested?

  • SQL file tests (main coverage, compared against Spark): a table of 17 relative and edge-case URLs queried for every part and a QUERY key in parse_url.sql, plus relative and error cases in parse_url_ansi.sql. CometSqlFileTestSuite parse_url passes locally (Spark 4.1 profile) and fails on main at the new queries.
  • Rust unit tests in parse_url.rs for the changed branch: test_relative_reference (a few relative references per part, expectations from Spark 4.1.1) and test_invalid_relative_reference (:, :foo, ://missing-scheme, //, http://). Both fail on main; all existing spark-expr tests pass, cargo fmt and clippy -D warnings are clean.
  • Differential check (local only, not committed) of the native UDFs against Spark 4.1.1 try_parse_url + java.net.URI validity on ~110k generated URLs and ~40k real-world relative page paths (each also tried with a host prepended): no change in results for absolute URLs except the http://-style inputs above; relative references now match Spark except the inputs covered by Native parse_url accepts URLs and registry-based hosts that java.net.URI treats differently #7330.
  • cargo bench --bench parse_url (main vs this branch): existing absolute-URL cases unchanged (all within ±4%). Scheme-less rows now do the normal extraction instead of returning early.

🤖 Generated with Claude Code

UPD: after review

  • Character validation now matches java.net.URI for relative and absolute URLs: ", \, ^, `, |, DEL and non-ASCII control or space characters, a % not followed by two hex digits, [ or ] in a hierarchical path, and a second # give INVALID_URL (ANSI) or NULL. This covers the character rows of Native parse_url accepts URLs and registry-based hosts that java.net.URI treats differently #7330, so the "Not in scope" paragraph above no longer applies to them.
  • Scheme names are validated (user_id:5, 1http://… are invalid), and a scheme with nothing after the : (http:, http:#f) is invalid.
  • Opaque URIs (mailto:x, urn:isbn:123, http:p?id=1) return only PROTOCOL and REF, as Spark does.
  • Still not in scope: registry-based authorities in extract_host (//ex_ample.com/p, the host rows of Native parse_url accepts URLs and registry-based hosts that java.net.URI treats differently #7330).
  • Tests: new test_parse_url_chars and test_parse_url_schemes tables in parse_url.sql and error cases in parse_url_ansi.sql. test_relative_reference and test_invalid_relative_reference were replaced by test_invisible_chars, since the SQL files cover the rest. The differential run was repeated. All ~40k real relative paths, with and without a host prepended, match Spark 4.1.1. On the ~110k generated URLs, no row that matched before stops matching, and matching rows go from 61k to 96k. The remaining differences are in the authority, which is Native parse_url accepts URLs and registry-based hosts that java.net.URI treats differently #7330.

@github-actions github-actions Bot added bug Something isn't working area:expressions Expression evaluation labels Nov 1, 2026
@valentinsoe
valentinsoe force-pushed the fix/parse-url-relative-urls branch from 5e6ef04 to 1b6f970 Compare November 2, 2026 10:17
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@valentinsoe
valentinsoe force-pushed the fix/parse-url-relative-urls branch from 1b6f970 to 778984c Compare November 2, 2026 10:19

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix. I compared a scratch copy of decompose_uri and parse_url_component from this head against java.net.URI on JDK 17, over a hand list and about 60k random short strings. The relative-reference results agree for every part, apart from host and character validity cases. I have not built Comet or run its tests.

The description notes that scheme-less inputs now reach the #7330 character gaps. That turns NULLs that matched Spark into wrong values. java.net.URI rejects /p|q, /p%zz, /p^q, /p"q, /p\q, /p[1], ?a=b|c and #a#b, so Spark returns NULL (or INVALID_URL under ANSI). main returned NULL for them, but this head returns, for example, PATH /p|q. |, " and a stray % are plausible in logged page paths, the #7385 case. How many of the ~40k real paths in your differential run fall in this group? Could the character checks for path, query, fragment and % escapes be tightened in this PR, or land first?

Not caused by this PR, and I did not see them in #7330: opaque URIs (mailto:x gives PATH x, Spark NULL) and http: or user_id:5 (Spark rejects them, Comet returns a PROTOCOL).

Comment on lines +184 to 194
// A value without a scheme is a relative reference; a leading ':' is an empty
// scheme name, which java.net.URI rejects ("Expected scheme name at index 0").
if scheme.is_none() && value.starts_with(':') {
return Err(invalid_url_err(value));
}

// java.net.URI rejects an empty authority ("//", "http://") unless a path,
// query or fragment follows it ("Expected authority").
if authority == Some("") && path.is_empty() && query.is_none() && fragment.is_none() {
return Err(invalid_url_err(value));
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

scheme.is_none() && is redundant, since decompose_uri only reports a scheme when the : is at an index above 0, so value.starts_with(':') is enough. In the second check, file is the path plus any ?query, so file.is_empty() && fragment.is_none() says the same thing. I read this from decompose_uri and have not run it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, both simplified as you suggested.

(':foo'),
('//'),
('http://'),
('http://example.com/r?to=https://y.com&id=9')

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The other tables in this file include a (NULL) row, but this one has none. Could you add (NULL) and ('')? The empty string returns before decompose_uri (PATH and FILE give '', the rest NULL), so it is a separate branch right next to the new code.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added ('') and (NULL), and (':') next to (':foo').

Comment on lines +841 to +850
fn test_invalid_relative_reference() {
for url in [":", ":foo", "://missing-scheme", "//", "http://"] {
for part in ["HOST", "PATH", "PROTOCOL", "FILE"] {
assert!(
parse_url_component(url, part, None).is_err(),
"{url} {part}"
);
}
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

://missing-scheme is already covered by test_malformed_with_scheme_separator and by the ANSI and parse_url_enabled.sql fixtures. :foo, // and http:// are in the new SQL table, and :foo and http:// are also in the ANSI file. Both new branches return before part is read, so the loop over four parts checks the same early return four times. Most rows of test_relative_reference repeat the SQL table too, which is compared with Spark directly. Could the Rust tests keep only what the SQL files do not cover?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed both. The SQL files cover those cases and compare with Spark directly. I kept one small Rust test, test_invisible_chars, for characters that are hard to see in a SQL file (DEL, NBSP, NEL, U+2028).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@valentinsoe

Copy link
Copy Markdown
Contributor Author

Thanks for checking this against java.net.URI directly.

You're right about the character gaps. In my run, 340 of the ~40k real page paths (0.85%) were invalid for java.net.URI, all because of a |. main returned NULL for them and this PR returned parsed values. All the other paths matched Spark.

I tightened the checks in this PR (81e5b3b):

  • ", \, ^, `, |, DEL and non-ASCII control or space characters are rejected anywhere in the value;
  • a % must be followed by two hex digits;
  • [ and ] are rejected in a hierarchical path, and stay allowed in a query, a fragment or an opaque part, as in java.net.URI;
  • a second # is rejected.

I also fixed the two cases you found outside this PR, since they sit in the same function:

  • a scheme name must be a letter followed by letters, digits, +, - or ., and must be followed by a non-empty scheme-specific part, so http:, http:#f and user_id:5 are rejected;
  • a scheme-specific part that does not start with / makes the URI opaque, as in java.net.URI: mailto:x#f gives PROTOCOL mailto and REF f, and NULL for the other parts.

The character checks apply to absolute URLs too, so they also cover the %zz and #a#b rows of #7330. The registry-based host rows there are still open.

With this commit, all of the ~40k real paths, with and without a host prepended, match Spark 4.1.1. On the ~110k generated URLs, no row that matched before stops matching, and the rows that match Spark go from 61k to 96k (out of 110k). The rest differ in the authority (registry-based hosts and IPv6 literals), which is #7330. The new cases are in the test_parse_url_chars and test_parse_url_schemes tables in parse_url.sql and in parse_url_ansi.sql.

@valentinsoe
valentinsoe requested a review from comphead November 4, 2026 19:08
@valentinsoe

Copy link
Copy Markdown
Contributor Author

@comphead thanks again for the detailed review. I addressed all the comments in 81e5b3b and the branch is up to date with main. Could you take another look when you have a moment? No rush, and happy to make further changes if needed.

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR backlog is huge and the maintainers don't have the bandwidth to review them all, so here is an automated review. Feel free to ignore it.

I found no blocking concerns. The change to parse_url_component in native/spark-expr/src/url_funcs/parse_url.rs matches java.net.URI for relative references, opaque URIs, scheme names and the character checks, which is what Spark's ParseUrl relies on.

To check this I copied decompose_uri, has_invalid_uri_chars and parse_url_component from this head into a scratch crate. I compared all eight parts (PROTOCOL, HOST, PATH, QUERY, REF, FILE, AUTHORITY, USERINFO) against new URI(...) on JDK 17 for 300k random strings built from tricky tokens such as :, //, %zz, [, |, NBSP, NEL, U+2028 and U+180E. Every difference was in the authority, either HOST of a registry-based authority, an unbalanced bracket pair, or USERINFO of an @ authority. That is the remaining part of #7330 and the PR already says so. The non-ASCII rule (control or whitespace rejected, everything else allowed) agreed with Character.isISOControl and Character.isSpaceChar on all of those inputs.

Against branch-1.2 the behavior changes are all intended and all move Comet toward Spark. Scheme-less values used to return NULL for every part and now return the parsed parts. http://, http:, user_id:5 and 1http://x used to return a PROTOCOL and now give INVALID_URL under ANSI or NULL otherwise. /r?to=http://x.com used to raise INVALID_URL and no longer does. Opaque URIs such as mailto:x used to return PATH x and now return only PROTOCOL and REF. I did not find an input where an answer that matched Spark on the release now differs.

The new has_invalid_uri_chars runs on every row, so I timed it against the old version on 100k URLs of about 110 bytes each. Both took roughly 80 ns per URL, so the stricter validation does not add measurable cost.

The SQL tables in parse_url.sql and parse_url_ansi.sql compare against Spark and cover the relative, character and scheme cases, and test_invisible_chars covers the characters a SQL file cannot show. I have no further requests.

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @valentinsoe

@andygrove
andygrove enabled auto-merge November 17, 2026 22:00
@andygrove
andygrove added this pull request to the merge queue Nov 17, 2026

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing the earlier points. In 81e5b3b2ca the scheme-less |, " and %zz paths are rejected again, the redundant scheme.is_none() and is_empty checks are gone, the SQL tables have (NULL) and ('') rows, and the Rust tests are down to test_invisible_chars. I copied the new parsing functions into a scratch program and compared them with new URI(...) on JDK 11 over about 900k random strings. PROTOCOL, PATH, QUERY, REF and FILE agree wherever both accept. The remaining differences are all in the authority: Comet accepts some strings with unbalanced or misplaced IPv6 brackets (for example //h[::1]x and //&&[]) that java.net.URI rejects, and HOST or USERINFO differ for registry-based authorities. That is the part of #7330 this PR leaves open. I have not built Comet or run its tests.

Merged via the queue into apache:main with commit 534c84b Nov 18, 2026
24 checks passed

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, great PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:expressions Expression evaluation bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native parse_url returns NULL for relative URLs (no scheme), Spark parses them

3 participants