Repository navigation
DX-101651: [C++][Gandiva] castVARCHAR(binary) validates UTF-8 - #158
vitaliibrodetskyi-prog wants to merge 3 commits into
Conversation
castVARCHAR(binary, int64) reinterpreted the bytes as VARCHAR and only
looked at them when it had to truncate, so invalid UTF-8 (for example the
output of UNBASE64('hello world!')) became a VARCHAR. Downstream string
functions such as length() then failed with "unexpected byte ... while
decoding utf8 string".
Validate the whole input before truncating, with a strict check (Unicode
Table 3-7: invalid lead bytes, bad or missing continuation bytes, overlong
forms, surrogates and code points above U+10FFFF) and raise
"Cannot cast VARBINARY to VARCHAR: input is not valid UTF-8". The function
is now flagged kCanReturnErrors so NULL rows, whose slots may hold arbitrary
bytes, are not validated.
The utf8 variant keeps the shared macro since Arrow guarantees its input.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Performance results (end-to-end Gandiva, baseline vs this PR's current head)Setup. Real Gandiva Noise. The two Arrow PRs touch different functions, so each is the other's control and its column should read 1.00x. It was 0.97x-1.03x everywhere except one sub-20 ns cell (1.12x); the two rounds differed by at most 16%, usually under 5%. Treat differences below ~10% as noise.
What the numbers mean
Not coveredx86-64/Linux (what production mostly runs on) was not measured; the ratios should be similar but absolute numbers will differ. Not measured: mixed value sizes within one batch, and values larger than 4 KB. Benchmark source (gbench.cc)#include <algorithm>
#include <chrono>
#include <cstdio>
#include <cstring>
#include <memory>
#include <random>
#include <string>
#include <vector>
#include "arrow/array/builder_binary.h"
#include "arrow/memory_pool.h"
#include "arrow/record_batch.h"
#include "arrow/status.h"
#include "arrow/type.h"
#include "gandiva/projector.h"
#include "gandiva/tree_expr_builder.h"
using gandiva::Projector;
using gandiva::TreeExprBuilder;
static constexpr int kRows = 4096;
static std::string Repeat(const std::string& unit, size_t bytes) {
std::string s;
while (s.size() < bytes) s += unit;
s.resize(bytes);
// do not cut a multi-byte character in half
while (!s.empty() && (static_cast<unsigned char>(s.back()) & 0xC0) == 0x80) s.pop_back();
if (!s.empty() && static_cast<unsigned char>(s.back()) >= 0xC0) s.pop_back();
return s;
}
static const char* kB64 = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
static std::string RandomBase64(std::mt19937& rng, int raw_bytes) {
std::string out;
int i = 0;
for (; i + 3 <= raw_bytes; i += 3)
for (int j = 0; j < 4; ++j) out += kB64[rng() % 64];
int rem = raw_bytes - i;
if (rem == 1) { out += kB64[rng() % 64]; out += kB64[(rng() % 16) * 4]; out += "=="; }
if (rem == 2) { out += kB64[rng() % 64]; out += kB64[rng() % 64]; out += kB64[(rng() % 16) * 4]; out += '='; }
return out;
}
template <typename Builder>
static std::shared_ptr<arrow::Array> Column(const std::vector<std::string>& values) {
Builder b;
for (const auto& v : values) (void)b.Append(v);
std::shared_ptr<arrow::Array> a;
(void)b.Finish(&a);
return a;
}
template <typename Builder>
static std::shared_ptr<arrow::Array> ColumnHalfNull(const std::vector<std::string>& values) {
Builder b;
for (size_t i = 0; i < values.size(); ++i) {
if (i % 2 == 1) (void)b.AppendNull(); else (void)b.Append(values[i]);
}
std::shared_ptr<arrow::Array> a;
(void)b.Finish(&a);
return a;
}
static std::string Unpadded(std::string s) { while (!s.empty() && s.back() == '=') s.pop_back(); return s; }
struct Case { std::string name; std::vector<std::string> values; };
static double MedianNsPerRow(const std::shared_ptr<Projector>& p,
const std::shared_ptr<arrow::RecordBatch>& batch) {
auto pool = arrow::default_memory_pool();
arrow::ArrayVector out;
for (int i = 0; i < 30; ++i) { out.clear(); (void)p->Evaluate(*batch, pool, &out); }
// calibrate repetitions to ~150 ms
auto t0 = std::chrono::steady_clock::now();
int calib = 20;
for (int i = 0; i < calib; ++i) { out.clear(); (void)p->Evaluate(*batch, pool, &out); }
double per = std::chrono::duration<double>(std::chrono::steady_clock::now() - t0).count() / calib;
int reps = std::max(5, static_cast<int>(0.15 / per));
std::vector<double> samples;
for (int r = 0; r < 9; ++r) {
auto s = std::chrono::steady_clock::now();
for (int i = 0; i < reps; ++i) { out.clear(); (void)p->Evaluate(*batch, pool, &out); }
double sec = std::chrono::duration<double>(std::chrono::steady_clock::now() - s).count();
samples.push_back(sec * 1e9 / (static_cast<double>(reps) * kRows));
}
std::sort(samples.begin(), samples.end());
return samples[samples.size() / 2];
}
int main(int argc, char** argv) {
std::string variant = argc > 1 ? argv[1] : "?";
std::mt19937 rng(42);
// ---- castVARCHAR(binary) ----
std::vector<Case> cast_cases = {
{"ascii_16B", {}}, {"ascii_256B", {}}, {"ascii_4096B", {}},
{"latin_64B", {}}, {"cjk_96B", {}}, {"emoji_64B", {}},
};
std::vector<std::pair<std::string, size_t>> units = {
{"abcdefgh", 16}, {"abcdefgh", 256}, {"abcdefgh", 4096},
{"h\xc3\xa9llo w\xc3\xb6rld ", 64}, {"\xe4\xb8\xad\xe6\x96\x87", 96}, {"\xf0\x9f\x98\x80", 64}};
for (size_t c = 0; c < cast_cases.size(); ++c) {
std::string v = Repeat(units[c].first, units[c].second);
cast_cases[c].values.assign(kRows, v);
}
auto bin_field = arrow::field("b", arrow::binary());
auto bin_schema = arrow::schema({bin_field});
auto node_b = TreeExprBuilder::MakeField(bin_field);
auto res_utf8 = arrow::field("r", arrow::utf8());
auto res_i32 = arrow::field("r", arrow::int32());
struct Expr { std::string name; gandiva::ExpressionPtr expr; };
auto len65536 = TreeExprBuilder::MakeLiteral(static_cast<int64_t>(65536));
auto len10 = TreeExprBuilder::MakeLiteral(static_cast<int64_t>(10));
auto cast_full = TreeExprBuilder::MakeFunction("castVARCHAR", {node_b, len65536}, arrow::utf8());
auto cast_trunc = TreeExprBuilder::MakeFunction("castVARCHAR", {node_b, len10}, arrow::utf8());
auto cast_len = TreeExprBuilder::MakeFunction("length", {cast_full}, arrow::int32());
std::vector<Expr> exprs = {
{"castVARCHAR(b,65536)", TreeExprBuilder::MakeExpression(cast_full, res_utf8)},
{"castVARCHAR(b,10)", TreeExprBuilder::MakeExpression(cast_trunc, res_utf8)},
{"length(castVARCHAR(b,65536))", TreeExprBuilder::MakeExpression(cast_len, res_i32)},
};
for (const auto& e : exprs) {
std::shared_ptr<Projector> p;
auto st = Projector::Make(bin_schema, {e.expr}, &p);
if (!st.ok()) { fprintf(stderr, "make failed: %s\n", st.ToString().c_str()); return 1; }
for (const auto& c : cast_cases) {
auto batch = arrow::RecordBatch::Make(bin_schema, kRows, {Column<arrow::BinaryBuilder>(c.values)});
printf("RESULT|%s|%s|%s|%s|%.2f\n", variant.c_str(), "cast", e.name.c_str(), c.name.c_str(),
MedianNsPerRow(p, batch));
fflush(stdout);
}
}
{
std::shared_ptr<Projector> p;
(void)Projector::Make(bin_schema, {exprs[0].expr}, &p);
auto batch = arrow::RecordBatch::Make(bin_schema, kRows, {ColumnHalfNull<arrow::BinaryBuilder>(cast_cases[4].values)});
printf("RESULT|%s|cast|castVARCHAR(b,65536)|latin_64B_half_NULL|%.2f\n", variant.c_str(), MedianNsPerRow(p, batch));
fflush(stdout);
}
// sanity: what does each build do with invalid bytes?
{
std::shared_ptr<Projector> p;
(void)Projector::Make(bin_schema, {exprs[0].expr}, &p);
auto batch = arrow::RecordBatch::Make(bin_schema, 2, {Column<arrow::BinaryBuilder>({"ok", "\x85\xe9\xa5"})});
arrow::ArrayVector out;
auto st = p->Evaluate(*batch, arrow::default_memory_pool(), &out);
printf("SANITY|%s|castVARCHAR_invalid_bytes|%s\n", variant.c_str(), st.ok() ? "accepted" : "rejected");
}
// ---- unbase64(utf8) ----
auto txt_field = arrow::field("s", arrow::utf8());
auto txt_schema = arrow::schema({txt_field});
auto node_s = TreeExprBuilder::MakeField(txt_field);
auto res_bin = arrow::field("r", arrow::binary());
auto unb = TreeExprBuilder::MakeFunction("unbase64", {node_s}, arrow::binary());
auto unb_expr = TreeExprBuilder::MakeExpression(unb, res_bin);
std::shared_ptr<Projector> up;
if (!Projector::Make(txt_schema, {unb_expr}, &up).ok()) { fprintf(stderr, "unbase64 make failed\n"); return 1; }
for (int raw : {8, 64, 1024, 8192}) {
std::vector<std::string> vals;
for (int i = 0; i < kRows; ++i) vals.push_back(RandomBase64(rng, raw));
auto batch = arrow::RecordBatch::Make(txt_schema, kRows, {Column<arrow::StringBuilder>(vals)});
printf("RESULT|%s|unbase64|unbase64(s)|b64_%dB_decoded|%.2f\n", variant.c_str(), raw, MedianNsPerRow(up, batch));
fflush(stdout);
}
{
std::vector<std::string> vals, unp;
for (int i = 0; i < kRows; ++i) { vals.push_back(RandomBase64(rng, 64)); unp.push_back(Unpadded(vals.back())); }
auto b1 = arrow::RecordBatch::Make(txt_schema, kRows, {Column<arrow::StringBuilder>(unp)});
printf("RESULT|%s|unbase64|unbase64(s)|b64_64B_unpadded|%.2f\n", variant.c_str(), MedianNsPerRow(up, b1));
auto b2 = arrow::RecordBatch::Make(txt_schema, kRows, {ColumnHalfNull<arrow::StringBuilder>(vals)});
printf("RESULT|%s|unbase64|unbase64(s)|b64_64B_half_NULL|%.2f\n", variant.c_str(), MedianNsPerRow(up, b2));
fflush(stdout);
}
{
auto batch = arrow::RecordBatch::Make(txt_schema, 2, {Column<arrow::StringBuilder>({"aGVsbG8=", "hello world!"})});
arrow::ArrayVector out;
auto st = up->Evaluate(*batch, arrow::default_memory_pool(), &out);
printf("SANITY|%s|unbase64_invalid_text|%s\n", variant.c_str(), st.ok() ? "accepted" : "rejected");
}
return 0;
}
|
…alidateUTF8 The hand-written validator walked multi-byte text about 2x slower than arrow::util::ValidateUTF8. Expose Arrow's validator as the external function gdv_fn_is_valid_utf8 and call it from castVARCHAR_binary_int64. Semantics are unchanged: the whole input is still validated, strictly (Unicode Table 3-7), before truncating. The stub initializes Arrow's UTF-8 tables first, because ValidateUTF8 accepts every input until they are. Measured end to end in a Gandiva projector, multi-byte values without truncation go from 3.3-5.8x to 2.0-3.0x of the unvalidated baseline. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Alternative (not in this PR): validate only the bytes that are returned when the cast truncatesWhen the cast truncates (
Measured (same end-to-end setup, multiple of the unvalidated baseline; this PR / alternative):
Trade-offs:
Diff of the alternative relative to this PR (
|
… in tests
UNBASE64('hello world!') decodes "hell" and stops at the space, which gives
85 E9 65, not 85 E9 A5. Both are invalid UTF-8 so the tests were valid, but the
comments named the wrong bytes.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
@akravchukdremio @lriggs if you get a chance, could you please review current changes? |
akravchukdremio
left a comment
There was a problem hiding this comment.
🟢 Verdict: No issues found. This implements the castVARCHAR fix agreed on DX-101651 (validate the whole input, then truncate, error on invalid UTF-8). Using Arrow's strict ValidateUTF8 also lines up with GuavaUtf8.isUtf8 in the Java cast, so both engines reject the same inputs.
Inspect first:
cpp/src/gandiva/gdv_function_stubs.cc: new external functiongdv_fn_is_valid_utf8, which callsInitializeUTF8()once before validating, plus itsi1mappingcpp/src/gandiva/function_registry_string.cc:kCanReturnErrorskeeps NULL rows from being validatedcpp/src/gandiva/precompiled/string_ops_test.cc: intended behaviour change,aa\xC3truncated to 2 characters now fails instead of returningaa
Change walkthrough (6 files)
| File | Change |
|---|---|
cpp/src/gandiva/function_registry_string.cc |
castVARCHAR(binary, int64) flagged kCanReturnErrors |
cpp/src/gandiva/gdv_function_stubs.cc |
New gdv_fn_is_valid_utf8 wrapping Arrow's ValidateUTF8, registered with the engine |
cpp/src/gandiva/gdv_function_stubs.h |
Declaration of gdv_fn_is_valid_utf8 |
cpp/src/gandiva/precompiled/string_ops.cc |
Standalone castVARCHAR_binary_int64: validates, then delegates to the utf8 truncation path |
cpp/src/gandiva/precompiled/string_ops_test.cc |
Valid and invalid UTF-8 cases; updated truncation/error assertions |
cpp/src/gandiva/tests/utf8_test.cc |
End-to-end projector test: NULL row with invalid bytes is skipped, invalid non-NULL input fails |
The failing CI jobs I checked failed on infrastructure (a missing Homebrew pkg-config@0.29.2 formula, a MinGW package-mirror timeout), not on this change.
AI-assisted analysis
|
Is this worth fixing giving the perf increases? What happens when you run this query with your changes? SELECT LENGTH(CAST(UNBASE64('hello world!') AS VARCHAR)) Is it possible to change the length function to return a better error message instead of having to check first? |
|
Thanks @lriggs, good questions. 1. Is this worth fixing given the perf increase? The cost is real, but only for
The validation itself is what the ticket decided on: the feedback on DX-101651 says I also compared this PR with the implementation sketched on the ticket (
It is also less strict: it accepts overlong forms ( Most of the time in this PR goes into Arrow's
Values under 32 bytes are unchanged (about 1.4x for ASCII 16 B). It accepts and rejects exactly the same inputs as this PR (15 byte sequences run through the cast, plus 18.8M inputs compared directly against Arrow's validator), but it vendors a roughly 3 MB source file, so it's a dependency decision and I haven't measured it on x86. Happy to do either (or both) if you think the cost matters. 2. What happens with It still fails, because
So #159 alone already gives this query a clear error, and it also makes 3. Can Yes, that's possible, and it was considered on the ticket ( |
Problem
castVARCHAR(binary, int64)reinterpreted raw bytes as VARCHAR and only inspected them when it had to truncate. Invalid UTF-8 (e.g.UNBASE64('hello world!')=85 E9 65) therefore became a VARCHAR, and later string functions failed:Change
castVARCHAR_binary_int64replaces the shared macro instantiation forbinary(theutf8variant is unchanged: Arrow guarantees its input).arrow::util::ValidateUTF8, exposed to the precompiled code as the external functiongdv_fn_is_valid_utf8(it relies on lookup tables, so it cannot be inlined into the bitcode). The stub callsInitializeUTF8()first:ValidateUTF8accepts every input until Arrow's tables are initialized.Cannot cast VARBINARY to VARCHAR: input is not valid UTF-8(same text as the planned Dremio Java cast).kCanReturnErrors, so Gandiva skips the call for NULL rows (their slots may hold arbitrary bytes).Verification
gandiva-precompiled-test: 135/135,gandiva-projector-test: 263/263 (TZ=UTC),gandiva-internals-test: 197/197. New tests:TestStringOps.TestCastVarcharFromBinaryValidatesUtf8(valid + 15 invalid shapes), updatedTestCastVarchar.gandiva-projector-testTestUtf8.TestCastVarCharFromBinary: end-to-end, a NULL row holding invalid bytes is not validated, invalid non-NULL input fails the batch.GuavaUtf8.isUtf8.castVARCHAR(b, 10)on a 4 KB ASCII value 20x (see the alternative below); insidelength(castVARCHAR(..))1.1-1.3x for ASCII and 1.6-1.7x for short multi-byte values. Re-measured after switching to Arrow's validator; tables and method in the performance comment. Not measured on x86-64/Linux.Alternative considered
Validating only the returned bytes when the cast truncates (cheaper truncation, slightly different semantics) was built and measured; see the comment below. It is not part of this PR.
Behaviour change
Binary that is not valid UTF-8 now fails the cast instead of producing an invalid VARCHAR. Companion Dremio change adds the same check to the Java cast so both engines agree.
Part of DX-101651. Dremio-side PRs follow once a build of this is available.
🤖 Generated with Claude Code
Additional info: related PRs
All PRs are part of DX-101651 ("LENGTH can't handle unicode characters"):
castVARCHAR(binary)validates UTF-8unbase64rejects invalid base64CAST(VARBINARY AS VARCHAR)LENGTH(VARBINARY)returns the size in bytesDremio counterpart: dremio/dremio#31344 adds the same check (same rule, same error message) to the Java cast, so Java and Gandiva agree on which inputs fail. Dremio only gets this change after
arrow.versioninoss/pom.xmlis bumped to a build containing this PR; until then apache#31344 is a draft and its Gandiva invalid-input test fails on purpose.Related, not dependent: dremio/arrow#159 changes a different function and can merge in any order. dremio/dremio#31345 uses Gandiva's existing
octet_length(binary)and does not need this PR.