Skip to content

test: add conformance tests over apache/iceberg-verification fixtures - #1144

Open
WZhuo wants to merge 1 commit into
apache:mainfrom
WZhuo:feat/iceberg-verification-phase1
Open

WZhuo wants to merge 1 commit into
apache:mainfrom
WZhuo:feat/iceberg-verification-phase1

Conversation

@WZhuo

@WZhuo WZhuo commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Phase 1 of #909: pin apache/iceberg-verification as a git submodule at testing/iceberg-verification, and add verification_test, which runs the types and schema surfaces of its table-spec fixtures.

Each fixture case becomes one GTest parameterized test named after <subdir>/<case-id>. The types cases go through TypeFromJson and the schema cases through SchemaFromJson. Each result is then projected into the fixture's decoded shape through public accessors and compared field by field. Every difference is reported with its path, for example fields[2].parent. The subscribed subdirectories are types/{primitive,nested,variant,geospatial} and schema/{core,unknown}.

The test is behind a new ICEBERG_BUILD_VERIFICATION_TESTS option, off by default. To run it:

git submodule update --init testing/iceberg-verification
cmake -S . -B build -DICEBERG_BUILD_VERIFICATION_TESTS=ON
cmake --build build --target verification_test
./build/src/iceberg/test/verification_test

ICEBERG_VERIFICATION_DIR points the test at another checkout of the fixtures.

Two things differ from the plan in #909:

  • If the option is on and the fixtures are missing, CMake fails at configure time. The plan had the tests skip instead, which would let a misconfigured run pass while testing nothing. With the option off by default, source tarballs are not affected.
  • There is no known-failures list. Every case that diverges from its fixture fails. Advisory cases (normative_level: should) are reported as skips. The test is not added to CI until all cases pass.

Currently 77 cases pass and 4 fail:

Case Issue
schema/unknown/reject-unknown-required #1141
types/primitive/decimal-9-2, types/primitive/decimal-9-2-spaced #1142
types/primitive/decimal-precision-over-38 #1143

Validation: built and ran verification_test locally on macOS with the results above. Configuring with a nonexistent ICEBERG_VERIFICATION_DIR fails as expected. pre-commit passes.

@wgtmac wgtmac left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally looks good! Please rebase to resolve ci failure.

Comment thread README.md
git submodule update --init testing/iceberg-verification
```

Pass `-DICEBERG_VERIFICATION_DIR=/path/to/iceberg-verification` to run against

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pin apache/iceberg-verification as a git submodule at
testing/iceberg-verification and add verification_test, which runs the
schema and types surfaces of its table-spec fixtures through
SchemaFromJson and TypeFromJson.

The test is built only with -DICEBERG_BUILD_VERIFICATION_TESTS=ON (off
by default) and is not run in CI yet. With the option on, CMake fails
at configure time if the fixtures are missing. Every case that diverges
from its fixture fails; advisory (normative_level: should) cases skip.

Four cases currently fail on implementation gaps:
- schema unknown/reject-unknown-required: SchemaFromJson accepts a
  required field of type unknown.
- types primitive/decimal-9-2 and decimal-9-2-spaced: ToJson writes
  decimal(P,S) without the space the fixtures expect.
- types primitive/decimal-precision-over-38: TypeFromJson throws from
  the DecimalType constructor instead of returning an error.
@WZhuo
WZhuo force-pushed the feat/iceberg-verification-phase1 branch from 276925e to 912ccb9 Compare November 20, 2026 03:52
@wgtmac

wgtmac commented Nov 20, 2026

Copy link
Copy Markdown
Member

@manuzhang @zhjwpku Do you want to take a look?

std::vector<VerificationCase> LoadCases(std::string_view surface,
std::span<const std::string_view> subdirs) {
std::vector<VerificationCase> cases;
if (!FixturesAvailable()) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Can we replace return and continue in this class with if ... else ...? We only need to return cases in the end.

load_error(e.what());
continue;
}
if (!fixture.contains(kCases) || !fixture[kCases].is_array()) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about empty fixture array?

load_error("no `cases` array");
continue;
}
for (auto& json : fixture[kCases]) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A null case entry would abort the test initialization. It would be better to explicitly check and report the error to identify the fixture file.

DiffJson(c.at(kDecoded), ProjectType(type), "", &diffs);
if (c.contains(kCanonical)) {
auto written = ToJson(type);
if (!written.has_value()) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Not sure why the negative case is checked. It doesn't go well with my human mind.

diffs->push_back(std::format("{}: unexpected {}", child, value.dump()));
}
}
return;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bear with me but someone just forgets there's such thing as if ... else if ... else. 😅

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants