Skip to content

DX-101651: [C++][Gandiva] unbase64 rejects invalid base64 input - #159

Open
vitaliibrodetskyi-prog wants to merge 6 commits into
dremio_27.0_23_19from
DX-101651-gandiva-strict-unbase64
Open

vitaliibrodetskyi-prog wants to merge 6 commits into
dremio_27.0_23_19from
DX-101651-gandiva-strict-unbase64

Conversation

@vitaliibrodetskyi-prog

@vitaliibrodetskyi-prog vitaliibrodetskyi-prog commented Nov 12, 2026 •

Copy link
Copy Markdown

Problem

unbase64 used arrow::util::base64_decode, which stops at the first character outside the alphabet and returns what it decoded so far:

SELECT UNBASE64('hello world!')   -- silently returns 85 E9 65 (decoded from "hell")

That truncated binary is what later breaks LENGTH(CAST(UNBASE64(..) AS VARCHAR)) (DX-101651). Spaces or line breaks in the middle of the input, stray = and a leftover single character were cut off the same way. Input that decoded to nothing also asked the arena for 0 bytes and failed with Could not allocate memory.

Change

  • Invalid base64 now fails with UNBASE64: input is not valid base64: <reason>, where the reason names the rule and position (unexpected character, data after padding, too many padding characters, incomplete last group).
  • Spaces, tabs, \n and \r are skipped, so 'aGVsbG8=\n' and line-wrapped input decode fully.
  • Padding stays optional (aGVsbG8 still decodes to hello), but when present it must complete the last group of four.
  • Non-zero trailing bits are accepted (ab== and ac== both give i), like Python's b64decode and commons-codec.
  • The output buffer is sized up front and never asks the arena for 0 bytes.
  • unbase64 is registered with kCanReturnErrors, so NULL rows (whose slots may hold arbitrary bytes) are not decoded.

Relationship to upstream

Apache Arrow fixed the same bug in apache#49660 (apacheGH-49614) and apache#49748. I did not take those, because they differ from what we need here:

Input before upstream this PR
aGVsbG8 (no padding) hello error hello
aGVsbG8=\n (trailing newline) hello error hello
aGVs bG8= (space inside) hel error hello
hello world!, aGVsbG8=aGVs, a garbage or empty error error
  • Upstream requires a length divisible by 4 and rejects whitespace, so it would break unpadded and newline-terminated base64 that decodes correctly today.
  • Upstream's unbase64 registration has no kCanReturnErrors, so a NULL row holding leftover bytes can fail the batch.
  • Standalone decode time (M4 Max, ns per value; 8 B / 64 B / 1 KB decoded): before 34 / 297 / 4508, upstream (both PRs) 27 / 158 / 2081, this PR 14 / 74 / 1402. Upstream is faster than the old decoder but still about 2x slower than this one for short values: it builds a std::string byte by byte, and the Gandiva stub then copies it into the arena.
  • Upstream also changes the shared base64_decode signature and its Parquet and Flight callers; this PR only touches Gandiva. When the fork is next upgraded to an Arrow with GH-49614: [C++] Report an error instead of silent truncation in base64_decode on invalid input apache/arrow#49660, the conflict in gdv_function_stubs.cc should be resolved in favour of this decoder.

Verification

  • gandiva-internals-test 200/200, gandiva-precompiled-test 134/134, gandiva-projector-test 264/264 (TZ=UTC).
  • TestBase64DecodeStrict (valid and invalid vectors, including those from GH-49614: [C++] Report an error instead of silent truncation in base64_decode on invalid input apache/arrow#49660), TestBase64DecodeErrorReason, TestBase64DecodeRoundTrip (every byte value and random data of every length up to 300 bytes, padded or not, with whitespace inserted), TestUnbase64NullSlotHoldsInvalidText (a NULL row whose slot holds hello world! is not decoded; fails without kCanReturnErrors), and TestUnbase64InvalidInput (whitespace input decodes, invalid row fails the batch).
  • Fuzzed against Python's validating b64decode: 300k cases, 0 mismatches. Valid input decodes byte-for-byte like before, and 200k random valid padded inputs agree between before, upstream and this PR.
  • End-to-end Gandiva: unbase64 takes 0.35-0.40x of the old time (about 2.5-2.9x faster) for 8 B to 8 KB values; details in the performance comment. Not measured on x86-64/Linux.

Behaviour change

Invalid base64 now fails the query instead of returning cut-off data. Valid base64 (padded or not, with a trailing line break) is unchanged. The Java/Hive fallback for UNBASE64 stays lenient, so a query's result can depend on which engine runs it (release note).

Related PRs (DX-101651)

Dremio picks this up with an arrow.version bump; the golden file TestStringFunctions.testFunctionsWithUnicodeCharacters.yaml then needs regenerating (tracked in apache#31344).

unbase64 relied on arrow::util::base64_decode, which stops at the first
character outside the alphabet and returns what it decoded so far, so
UNBASE64('hello world!') silently produced the 3 bytes 85 E9 A5 instead of
failing. Every other malformed input (stray '=', leftover single character,
line breaks) was accepted the same way.

Decode with a strict RFC 4648 decoder instead and raise
"UNBASE64: input is not valid base64". Padding stays optional, as before, but
when present it must complete the final group of four characters.

The output length is computed up front, and empty output returns "" without
calling the arena. Previously a 0-byte allocation could return nullptr, which
was reported as "Could not allocate memory" (e.g. for input made only of
non-ASCII characters).

unbase64 is now flagged kCanReturnErrors so NULL rows, whose slots may hold
arbitrary bytes, are not decoded.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@vitaliibrodetskyi-prog

vitaliibrodetskyi-prog commented Nov 12, 2026 •

Copy link
Copy Markdown
Author

Performance results (end-to-end Gandiva, baseline vs this PR's current head)

Setup. Real Gandiva Projector evaluating unbase64 over a 4096-row batch (a typical batch size); every row holds the same value unless noted. Two libgandiva builds: baseline f21f62c and #159 at 7d0bc69 (Release, LLVM 21). Numbers are wall-clock ns per row for Evaluate (output allocation included, JIT compile excluded): median of 9 timed runs, averaged over 2 interleaved rounds (baseline, #159, baseline, #159), single thread, Apple M4 Max. Behaviour was verified on each build (the baseline accepts invalid base64, #159 rejects it).

Noise. This PR does not touch castVARCHAR, so the castVARCHAR and length(castVARCHAR(..)) benchmarks run on this build are the control: 19 cells at 0.95x-1.07x of baseline (3 outside 0.97x-1.03x). Treat differences below ~10% as noise. This run measured the unbase64 rows only, with 2 interleaved rounds.

unbase64(s) over a utf8 column of valid base64 text

Input baseline ns/row #159 ns/row ratio
8 B decoded (12 chars) 87.2 33.8 0.39x
64 B decoded (88 chars) 806.9 305.3 0.38x
64 B decoded, no = padding 809.6 297.3 0.37x
64 B decoded, every other row NULL 399.5 140.7 0.35x
1 KB decoded (1368 chars) 12666.2 5021.5 0.40x
8 KB decoded (10924 chars) 99362.7 40078.5 0.40x

What the numbers mean

  • The strict decoder is about 2.5-2.9x faster than before, at every size measured (8 B to 8 KB decoded, padded or not, with or without NULL rows). Valid input decodes identically (unit tests, plus fuzzing against Python's validating decoder, see the description). So this PR adds no cost; it removes some.
  • Why: the previous path called arrow::util::base64_decode, which builds a std::string per row and looks every character up with base64_chars.find(), a linear search of the alphabet. The new decoder computes the output length up front, uses a branch-based value lookup and writes straight into the arena buffer.
  • The extra checks (alphabet, whitespace, padding, length) are not measurable on top of that: whitespace and = are handled only in the slow path, valid characters take the same path as before. Re-measured after adding whitespace skipping: 0.35x-0.40x, same as without it. kCanReturnErrors (NULL rows are skipped) adds one validity check per row; with half the rows NULL the cost drops in proportion (the NULL rows are free).

Not covered

x86-64/Linux was not measured. Invalid input is not part of the benchmark: it fails the batch, so there is no steady-state throughput to compare. The benchmark program is in the collapsed section of the performance comment on #158.

The strict decoder rejected whitespace, but output of the base64 tool ends
with a line break and RFC 2045 (MIME) allows line breaks anywhere, so inputs
such as 'aGVsbG8=\n' that decoded correctly before would have started to fail.

Skip ' ', '\t', '\n' and '\r' while decoding. Everything else that is not in the
alphabet, data after padding, and malformed padding or length are still errors
(whitespace is ignored when checking them).

Padding is now handled in the same loop, so the output buffer is sized with an
upper bound (3/4 of the input, at least 1 byte) instead of exactly; unrelated to
speed, valid-character handling is unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@vitaliibrodetskyi-prog

Copy link
Copy Markdown
Author

@akravchukdremio @lriggs if you get a chance, could you please review current changes?

@akravchukdremio akravchukdremio left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Verdict: No issues found. The strict decoder is correct, the output buffer bound is safe, and kCanReturnErrors keeps NULL rows away from the decoder.

Inspect first:

  • cpp/src/gandiva/gdv_function_stubs.cc: new decoder loop and the post-loop padding/length checks
  • cpp/src/gandiva/function_registry_string.cc: kCanReturnErrors on unbase64 (needed, since the decoder now reads NULL slots' bytes as errors otherwise)
Change walkthrough (4 files)
File Change
cpp/src/gandiva/gdv_function_stubs.cc Replaces truncating arrow::util::base64_decode with a strict decoder that skips whitespace, has optional but validated padding, and never makes a 0-byte arena allocation
cpp/src/gandiva/function_registry_string.cc unbase64 gains kCanReturnErrors; unrelated clang-format reflow of encrypt/decrypt entries
cpp/src/gandiva/gdv_function_stubs_test.cc New TestBase64DecodeStrict with valid and invalid vectors
cpp/src/gandiva/tests/projector_test.cc New TestUnbase64InvalidInput: NULL row not decoded, whitespace input decodes, invalid row fails the batch

Checked:

  • The buffer bound max(in_len*3/4, 1) covers every write, including bytes written before invalid input is rejected after the loop. I compiled the decoder loop standalone under ASan/UBSan with a bounds abort and fuzzed it with 200k inputs against Python's b64decode(validate=True) (whitespace stripped, missing padding added): 0 mismatches, no overruns.
  • kCanReturnErrors makes the LLVM generator wrap the call in a validity check, so NULL rows never reach the decoder.

Non-blocking notes:

  • Invalid input now fails the batch, while Dremio's Java/Hive fallback for UNBASE64 stays lenient, so a query's result can depend on which engine runs it. The description already calls this out; worth a line in the release notes.
  • Non-zero trailing bits are accepted (ab== and ac== both decode to i). That matches Python and commons-codec, so it's fine, but "strict RFC 4648" slightly overstates it.
  • A few edge cases could be pinned in tests: lone = / ==, a=, and rejected \v / \f.

AI-assisted analysis

Cover a lone padding character, padding without data, a single data
character followed by padding, and vertical tab / form feed, which are
not skipped like space, tab, CR and LF.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

@akravchukdremio akravchukdremio left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 9b7bab2. The new commit only adds invalid-input test cases (lone = / ==, a=, \v, \f), and the decoder is unchanged since my earlier review.

  • Built locally (macOS arm64, Debug, LLVM 16): gandiva-internals-test 198/198 and gandiva-projector-test 263/263 (TZ=UTC) pass, including TestBase64DecodeStrict and TestUnbase64InvalidInput. No warnings in the touched files.
  • The CI failures on this head are infrastructure: MinIO download 410 Gone in the Docker builds, LLVM not found on MinGW, Homebrew/vcpkg breakage. CI never got to building or running Gandiva.
  • An unbase64 in an untaken IF/CASE branch is not evaluated inside Gandiva (BuildIfElse emits each branch in its own block), so the new error only fires on rows that actually reach it.

Release note: invalid base64 now fails the query in Gandiva, while the Java fallback for UNBASE64 stays lenient.


AI-assisted analysis

@akravchukdremio akravchukdremio left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code looks good

The error now names the rule that failed (unexpected character, data
after padding, too many padding characters, incomplete group) and its
position. Also test the vectors from apache#49660, except for the
unpadded input that is accepted here.
vitaliibrodetskyi-prog added 2 commits November 18, 2026 12:40
…formatting

Add a test that encodes every byte value and random data of every length
up to 300 bytes, with and without padding and with whitespace inserted,
and checks that unbase64 returns the original bytes. Restore the
encrypt/decrypt registry entries that an earlier commit reflowed.
… text

The existing NULL-row test builds its array with an empty NULL slot, so it
passes even without kCanReturnErrors. Put "hello world!" in the NULL
slot: the batch only succeeds because the NULL row is not decoded.
@vitaliibrodetskyi-prog

Copy link
Copy Markdown
Author

@akravchukdremio could you please review once again?
@dchirkov thanks for checking upstream. I looked at taking apache#49660 and apache#49748 into the fork. They apply cleanly, but I kept our decoder, for three reasons:

  • Compatibility. Upstream requires a length divisible by 4 and rejects whitespace, so unpadded base64 (aGVsbG8) and base64 ending in a newline would start failing. Both decode correctly in the fork today.
  • Speed. In a standalone comparison (decoded size → ns per value): 64 B: 297 before, 158 upstream with both PRs, 74 ours; 1 KB: 4508 / 2081 / 1402; 8 KB: about 39,500 / 15,500 / 12,700. Upstream is faster than what we have today, but ours is still about 2x faster for short values and about 1.2–1.5x for 1 KB and larger.
  • NULL rows. Upstream's unbase64 registration has no kCanReturnErrors, so a NULL row with leftover bytes can fail the batch. Here NULL rows are skipped, and a new test covers it (a NULL slot holding hello world!; it fails without the flag).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants