Repository navigation
DX-101651: [C++][Gandiva] unbase64 rejects invalid base64 input - #159
vitaliibrodetskyi-prog wants to merge 6 commits into
Conversation
unbase64 relied on arrow::util::base64_decode, which stops at the first
character outside the alphabet and returns what it decoded so far, so
UNBASE64('hello world!') silently produced the 3 bytes 85 E9 A5 instead of
failing. Every other malformed input (stray '=', leftover single character,
line breaks) was accepted the same way.
Decode with a strict RFC 4648 decoder instead and raise
"UNBASE64: input is not valid base64". Padding stays optional, as before, but
when present it must complete the final group of four characters.
The output length is computed up front, and empty output returns "" without
calling the arena. Previously a 0-byte allocation could return nullptr, which
was reported as "Could not allocate memory" (e.g. for input made only of
non-ASCII characters).
unbase64 is now flagged kCanReturnErrors so NULL rows, whose slots may hold
arbitrary bytes, are not decoded.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Performance results (end-to-end Gandiva, baseline vs this PR's current head)Setup. Real Gandiva Noise. This PR does not touch
What the numbers mean
Not coveredx86-64/Linux was not measured. Invalid input is not part of the benchmark: it fails the batch, so there is no steady-state throughput to compare. The benchmark program is in the collapsed section of the performance comment on #158. |
The strict decoder rejected whitespace, but output of the base64 tool ends with a line break and RFC 2045 (MIME) allows line breaks anywhere, so inputs such as 'aGVsbG8=\n' that decoded correctly before would have started to fail. Skip ' ', '\t', '\n' and '\r' while decoding. Everything else that is not in the alphabet, data after padding, and malformed padding or length are still errors (whitespace is ignored when checking them). Padding is now handled in the same loop, so the output buffer is sized with an upper bound (3/4 of the input, at least 1 byte) instead of exactly; unrelated to speed, valid-character handling is unchanged. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
@akravchukdremio @lriggs if you get a chance, could you please review current changes? |
akravchukdremio
left a comment
There was a problem hiding this comment.
🟢 Verdict: No issues found. The strict decoder is correct, the output buffer bound is safe, and kCanReturnErrors keeps NULL rows away from the decoder.
Inspect first:
cpp/src/gandiva/gdv_function_stubs.cc: new decoder loop and the post-loop padding/length checkscpp/src/gandiva/function_registry_string.cc:kCanReturnErrorsonunbase64(needed, since the decoder now reads NULL slots' bytes as errors otherwise)
Change walkthrough (4 files)
| File | Change |
|---|---|
cpp/src/gandiva/gdv_function_stubs.cc |
Replaces truncating arrow::util::base64_decode with a strict decoder that skips whitespace, has optional but validated padding, and never makes a 0-byte arena allocation |
cpp/src/gandiva/function_registry_string.cc |
unbase64 gains kCanReturnErrors; unrelated clang-format reflow of encrypt/decrypt entries |
cpp/src/gandiva/gdv_function_stubs_test.cc |
New TestBase64DecodeStrict with valid and invalid vectors |
cpp/src/gandiva/tests/projector_test.cc |
New TestUnbase64InvalidInput: NULL row not decoded, whitespace input decodes, invalid row fails the batch |
Checked:
- The buffer bound
max(in_len*3/4, 1)covers every write, including bytes written before invalid input is rejected after the loop. I compiled the decoder loop standalone under ASan/UBSan with a bounds abort and fuzzed it with 200k inputs against Python'sb64decode(validate=True)(whitespace stripped, missing padding added): 0 mismatches, no overruns. kCanReturnErrorsmakes the LLVM generator wrap the call in a validity check, so NULL rows never reach the decoder.
Non-blocking notes:
- Invalid input now fails the batch, while Dremio's Java/Hive fallback for
UNBASE64stays lenient, so a query's result can depend on which engine runs it. The description already calls this out; worth a line in the release notes. - Non-zero trailing bits are accepted (
ab==andac==both decode toi). That matches Python and commons-codec, so it's fine, but "strict RFC 4648" slightly overstates it. - A few edge cases could be pinned in tests: lone
=/==,a=, and rejected\v/\f.
AI-assisted analysis
Cover a lone padding character, padding without data, a single data character followed by padding, and vertical tab / form feed, which are not skipped like space, tab, CR and LF. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
akravchukdremio
left a comment
There was a problem hiding this comment.
Approving at 9b7bab2. The new commit only adds invalid-input test cases (lone = / ==, a=, \v, \f), and the decoder is unchanged since my earlier review.
- Built locally (macOS arm64, Debug, LLVM 16):
gandiva-internals-test198/198 andgandiva-projector-test263/263 (TZ=UTC) pass, includingTestBase64DecodeStrictandTestUnbase64InvalidInput. No warnings in the touched files. - The CI failures on this head are infrastructure: MinIO download
410 Gonein the Docker builds, LLVM not found on MinGW, Homebrew/vcpkg breakage. CI never got to building or running Gandiva. - An
unbase64in an untakenIF/CASEbranch is not evaluated inside Gandiva (BuildIfElseemits each branch in its own block), so the new error only fires on rows that actually reach it.
Release note: invalid base64 now fails the query in Gandiva, while the Java fallback for UNBASE64 stays lenient.
AI-assisted analysis
The error now names the rule that failed (unexpected character, data after padding, too many padding characters, incomplete group) and its position. Also test the vectors from apache#49660, except for the unpadded input that is accepted here.
…formatting Add a test that encodes every byte value and random data of every length up to 300 bytes, with and without padding and with whitespace inserted, and checks that unbase64 returns the original bytes. Restore the encrypt/decrypt registry entries that an earlier commit reflowed.
… text The existing NULL-row test builds its array with an empty NULL slot, so it passes even without kCanReturnErrors. Put "hello world!" in the NULL slot: the batch only succeeds because the NULL row is not decoded.
|
@akravchukdremio could you please review once again?
|
Problem
unbase64usedarrow::util::base64_decode, which stops at the first character outside the alphabet and returns what it decoded so far:That truncated binary is what later breaks
LENGTH(CAST(UNBASE64(..) AS VARCHAR))(DX-101651). Spaces or line breaks in the middle of the input, stray=and a leftover single character were cut off the same way. Input that decoded to nothing also asked the arena for 0 bytes and failed withCould not allocate memory.Change
UNBASE64: input is not valid base64: <reason>, where the reason names the rule and position (unexpected character, data after padding, too many padding characters, incomplete last group).\nand\rare skipped, so'aGVsbG8=\n'and line-wrapped input decode fully.aGVsbG8still decodes tohello), but when present it must complete the last group of four.ab==andac==both givei), like Python'sb64decodeand commons-codec.unbase64is registered withkCanReturnErrors, so NULL rows (whose slots may hold arbitrary bytes) are not decoded.Relationship to upstream
Apache Arrow fixed the same bug in apache#49660 (apacheGH-49614) and apache#49748. I did not take those, because they differ from what we need here:
aGVsbG8(no padding)hellohelloaGVsbG8=\n(trailing newline)hellohelloaGVs bG8=(space inside)helhellohello world!,aGVsbG8=aGVs,aunbase64registration has nokCanReturnErrors, so a NULL row holding leftover bytes can fail the batch.std::stringbyte by byte, and the Gandiva stub then copies it into the arena.base64_decodesignature and its Parquet and Flight callers; this PR only touches Gandiva. When the fork is next upgraded to an Arrow with GH-49614: [C++] Report an error instead of silent truncation in base64_decode on invalid input apache/arrow#49660, the conflict ingdv_function_stubs.ccshould be resolved in favour of this decoder.Verification
gandiva-internals-test200/200,gandiva-precompiled-test134/134,gandiva-projector-test264/264 (TZ=UTC).TestBase64DecodeStrict(valid and invalid vectors, including those from GH-49614: [C++] Report an error instead of silent truncation in base64_decode on invalid input apache/arrow#49660),TestBase64DecodeErrorReason,TestBase64DecodeRoundTrip(every byte value and random data of every length up to 300 bytes, padded or not, with whitespace inserted),TestUnbase64NullSlotHoldsInvalidText(a NULL row whose slot holdshello world!is not decoded; fails withoutkCanReturnErrors), andTestUnbase64InvalidInput(whitespace input decodes, invalid row fails the batch).b64decode: 300k cases, 0 mismatches. Valid input decodes byte-for-byte like before, and 200k random valid padded inputs agree between before, upstream and this PR.unbase64takes 0.35-0.40x of the old time (about 2.5-2.9x faster) for 8 B to 8 KB values; details in the performance comment. Not measured on x86-64/Linux.Behaviour change
Invalid base64 now fails the query instead of returning cut-off data. Valid base64 (padded or not, with a trailing line break) is unchanged. The Java/Hive fallback for
UNBASE64stays lenient, so a query's result can depend on which engine runs it (release note).Related PRs (DX-101651)
castVARCHAR(binary)validates UTF-8 (independent, can merge in any order)LENGTH(VARBINARY)returns the size in bytesDremio picks this up with an
arrow.versionbump; the golden fileTestStringFunctions.testFunctionsWithUnicodeCharacters.yamlthen needs regenerating (tracked in apache#31344).