Repository navigation
GH-44949: [C++][JSON] Reject duplicate field names in explicit schemas - #55641
AmanBoronbaev wants to merge 2 commits into
Conversation
|
|
|
Regarding the initialization-complexity question:
This does add a constant-factor cost: Code at the submitted commit:
|
Signed-off-by: Aman <149809343+AmanBoronbaev@users.noreply.github.com>
|
Follow-up on initialization overhead: updated in ddb2151 to reuse the lookup already performed by I also measured parser construction/destruction with fixed-length unique field names and schema construction outside the timed loop. Local results below are medians of seven batches after one warm-up batch, with
Environment: Intel i5-12450H, Clang 22.1.8, Arrow Debug library ( Validation after the update: all 254 C++ JSON tests passed, including duplicate names at root/nested/list levels and valid schemas; clang-format, cpplint and the Standalone reproduction driver#include <chrono>
#include <iomanip>
#include <iostream>
#include <memory>
#include <string>
#include "arrow/json/options.h"
#include "arrow/json/parser.h"
#include "arrow/type.h"
int main() {
std::cout << "fields,repeat,iterations,ns_per_init,ns_per_field\n";
for (int count : {10, 100, 1000, 10000}) {
arrow::FieldVector fields;
for (int i = 0; i < count; ++i) {
// Fixed-length names keep hashing cost comparable across schema widths.
auto suffix = std::to_string(i);
fields.push_back(arrow::field("field_" + std::string(8 - suffix.size(), '0') + suffix,
arrow::int64()));
}
auto options = arrow::json::ParseOptions::Defaults();
options.explicit_schema = arrow::schema(fields);
options.unexpected_field_behavior = arrow::json::UnexpectedFieldBehavior::Error;
int iterations = 100000 / count;
for (int repeat = -1; repeat < 7; ++repeat) {
auto start = std::chrono::steady_clock::now();
for (int i = 0; i < iterations; ++i) {
std::unique_ptr<arrow::json::BlockParser> parser;
auto status = arrow::json::BlockParser::Make(options, &parser);
if (!status.ok()) { std::cerr << status << '\n'; return 1; }
}
double ns = std::chrono::duration<double, std::nano>(
std::chrono::steady_clock::now() - start).count() / iterations;
if (repeat >= 0) std::cout << count << ',' << repeat << ',' << iterations << ','
<< std::fixed << std::setprecision(1) << ns << ',' << ns/count << '\n';
}
}
}
Build the Arrow JSON shared library at each revision with the same configuration. Compile this driver against that checkout's |
Rationale for this change
Fixes #44949.
When an explicit JSON schema contains two fields named
foo, the parser silently reuses the first field's index. The parsed fields then no longer match the explicit schema, and the reader can report an unrelated conversion error. Reject duplicate sibling field names with anInvalidstatus that identifies the duplicate.What changes are included in this PR?
Are these changes tested?
Built from main commit
55803660f015f9945fe1b9bc84f0c138ae70ce98with Clang 22.1.8, Debug assertions, andBUILD_WARNING_LEVEL=CHECKIN(-Werror).arrow-json-testexecutable passed all 254 tests across 60 suites, including the 8 added parameterized test instances.clang-format 18.1.8 --dry-run --Werror,cpplint 1.6.1 --quiet --verbose=2, andgit diff --checkpassed for all changed files.The complete Arrow CI matrix and Python bindings were not built or run locally.
Are there any user-facing changes?
JSON reading now reports
Invalid: JSON parse error: duplicate field 'foo' in explicit schemafor duplicate sibling names. General Arrow schemas and CSV handling are unchanged. There are no public API signature changes.