Skip to content

GH-3043: Ignore V1 level encodings for omitted streams - #3867

Merged
wgtmac merged 1 commit into
apache:masterfrom
skyon0522-ai:codex/v1-zero-level-streams
Oct 29, 2026
Merged

wgtmac merged 1 commit into
apache:masterfrom
skyon0522-ai:codex/v1-zero-level-streams

Conversation

@skyon0522-ai

Copy link
Copy Markdown
Contributor

Closes #3043.

Rationale for this change

The attached nanoparquet file uses PLAIN in the V1 repetition/definition
encoding fields for flat required columns. Neither level stream is present.
readPageV1 nevertheless selects a reader from each header encoding, which
can consume value bytes or fail at ValuesReader.readInteger before the
first record is returned. The format specification
specifies that these zero-maximum level streams are omitted.

What changes are included in this PR?

Select the existing ZeroIntegerValuesReader independently when the
corresponding maximum level is zero. Keep the existing encoding dispatch
for nonzero levels. The regression uses the upstream V1 writer to generate
two pages, replaces only unused level-encoding fields, and checks exact
values and levels for required, optional and repeated columns.

Are these changes tested?

Validation on Linux (Ubuntu 24.04, Java 17.0.20.1, Maven 3.9.11):

  • The new parameterized cases produced four errors on unchanged production
    code. The two controls and seven existing cases passed. All 13 passed
    with the fix.
  • The full parquet-column reactor test run passed all 1,094 tests.
  • A separate 92-test Hadoop run passed, including reading all 32 rows of
    the issue attachment through both reader APIs and a generated control.
  • Spotless checks for the two changed files, normal RAT checks across
    eight reactor modules, and git diff --check passed.

The entire root reactor, native Windows and macOS were not tested.

Are there any user-facing changes?

Files such as the issue attachment become readable when a V1 header uses
PLAIN for an omitted level stream.

OpenAI Codex assisted with the implementation and tests.

Use the existing zero reader when a column's maximum repetition or
definition level is zero. Such a column has no corresponding encoded
level stream, so its unused header encoding must not select a physical
value reader or consume value bytes.

Preserve encoding dispatch for nonzero levels and add synthetic V1
regressions for required, optional and repeated columns across two pages.

Generated-by: OpenAI Codex
@dossett

dossett commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

Here's how I understand the issue (full disclosure: arrived at with the help of a conversation with codex, but these words are mine). The v1 specification requires definition_level_encoding and repetition_level_encoding but sometimes they aren't actually relevant. This change first looks to the schema to determine if they're relevant and then uses them as opposed to using them by default, which can lead to issues like this.

Based on that understanding I am +1 (non-binding).


Here's a minimum reproduction of the issue furnished by codex.

@Test
public void requiredDoubleIgnoresAbsentRepetitionLevelEncoding() throws Exception {
  ColumnDescriptor column = MessageTypeParser
      .parseMessageType("message test { required double foo; }")
      .getColumns()
      .get(0);

  byte[] data = ByteBuffer.allocate(Double.BYTES)
      .order(ByteOrder.LITTLE_ENDIAN)
      .putDouble(12.5)
      .array();

  DataPageV1 page = new DataPageV1(
      BytesInput.from(data),
      1,
      data.length,
      new DoubleStatistics(),
      Encoding.PLAIN, // repetition levels: absent despite this label
      Encoding.RLE,   // definition levels: also absent
      Encoding.PLAIN);

  ColumnReader reader = new ColumnReaderImpl(
      column,
      new MemPageReader(1, List.<DataPage>of(page).iterator(), null),
      new PrimitiveConverter() {},
      VersionParser.parse(Version.FULL_VERSION));

  assertThat(reader.getDouble()).isEqualTo(12.5);
}

@wgtmac wgtmac left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree this fix makes sense. Thanks @skyon0522-ai for fixing this and @dossett for the review and analysis!

@wgtmac
wgtmac merged commit 1193f42 into apache:master Oct 29, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cannot read parquet file that was generated from nanoparquet

3 participants