Skip to content

GH-3877: Add FINITE_VECTOR logical type support - #3876

Open
divjotarora wants to merge 19 commits into
apache:masterfrom
divjotarora:vector
Open

divjotarora wants to merge 19 commits into
apache:masterfrom
divjotarora:vector

Conversation

@divjotarora

@divjotarora divjotarora commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Rationale for this change

The FINITE_VECTOR proposal defines fixed-size vectors of non-null, finite elements on canonical LIST storage, so applications can express vector semantics while retaining existing Parquet storage and interoperability. This change brings parquet-java support in line with the proposal.

What changes are included in this PR?

Adds FINITE_VECTOR schema annotations and mandatory write validation, validates stored levels and elements during reads, and preserves LIST interoperability across existing model readers.

Are these changes tested?

The affected Maven suites for FINITE_VECTOR schemas, writing, reading, adapters, and filtered reads passed, and Spotless passed for all touched modules.

Closes #3877

@divjotarora
divjotarora marked this pull request as ready for review October 22, 2026 10:39
@divjotarora divjotarora changed the title MINOR: Add VECTOR logical type support GH-3877: Add VECTOR logical type support Oct 22, 2026
@divjotarora divjotarora changed the title GH-3877: Add VECTOR logical type support GH-3877: Add FINITE_VECTOR logical type support Nov 2, 2026

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The biggest issue is around the validation. It feels to me that a lot of visitor that are meant to do other things, are now also validating if it is a valid FINITE_VECTOR. I left some other comments as well. Let me know what you think

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eventually, these should go in https://github.com/apache/parquet-testing/

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's definitely the goal here

@divjotarora
divjotarora requested a review from Fokko November 4, 2026 15:36

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @divjotarora for iterating on this, I think we're getting close 👍 Now, my main concern is the performance penalty that we're having because of the validation.

@Fokko

Fokko commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Another thing, I think we don't need to collect that much statistics for the vector:

  • min/max don't add much value
  • Bloom filters even less
  • null-count does not make much sense, since the fields are required

We can also do this in a separate PR

Co-authored-by: Isaac <no-reply@databricks.com>
divjotarora and others added 2 commits November 11, 2026 14:17
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for iterating on this! I ran a benchmark comparing FINITE_VECTOR(n) against a plain LIST holding the same data (4M floats, uncompressed, no dictionary, float[] in/out through a minimal WriteSupport/ReadSupport so the object model doesn't dominate). readPages reads row groups and pages without record assembly, which is where the count checks run:

ms/op dim V1 vector / list V2 vector / list
write 8 94.9 / 104.0 90.5 / 98.6
write 128 75.4 / 76.4 78.2 / 82.4
write 1024 70.5 / 71.2 71.9 / 82.0
readRecords 8 74.9 / 71.8 50.7 / 60.4
readRecords 128 66.3 / 67.7 49.3 / 49.3
readRecords 1024 43.6 / 44.5 37.8 / 37.1
readPages 8 3.55 / 3.42 3.39 / 3.38
readPages 128 3.37 / 3.33 3.29 / 3.51
readPages 1024 3.44 / 3.30 3.43 / 3.23

There's no measurable difference in either direction (single fork, so the swings both ways are run-to-run noise), so removing the per-value validation took care of my performance concern 👍 I can open the benchmark as a separate PR.

My main remaining point is about the reader-side count checks (see inline): now that we delegate validation to the engine, they only catch some invalid files. The rest are small things.

VectorColumn vector = vectorColumns.get(column.getPath());
if (vector == null) return;
org.apache.parquet.column.statistics.Statistics<?> stats = column.getStatistics();
long nulls = stats != null && stats.isNumNullsSet() ? stats.getNumNulls() : -1;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This turns null_count from the statistics into a hard read failure. Statistics are optional, and we've had writers producing incorrect ones before, so a file with valid data but a bad null_count becomes unreadable. One more reason to drop the count checks (see above).

}
}

public static class FiniteVectorLogicalTypeAnnotation extends ListLogicalTypeAnnotation {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The spec says every non-null vector MUST have exactly num_elements elements and every element MUST be finite. Since parquet-java doesn't enforce either (testVectorValuesAreNotValidated writes NaN/Inf), could we add a Javadoc here saying so? Then users know it's up to them to write valid vectors.

@Override
public boolean equals(Object obj) {
return obj instanceof ListLogicalTypeAnnotation;
return obj instanceof ListLogicalTypeAnnotation && !(obj instanceof FiniteVectorLogicalTypeAnnotation);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this way the parent needs to know about its subclass. How about:

Suggested change
return obj instanceof ListLogicalTypeAnnotation && !(obj instanceof FiniteVectorLogicalTypeAnnotation);
return obj != null && obj.getClass() == ListLogicalTypeAnnotation.class;

Comment on lines 270 to +275

@Override
public Optional<Converter> visit(
LogicalTypeAnnotation.FiniteVectorLogicalTypeAnnotation vectorLogicalType) {
return visit(LogicalTypeAnnotation.listType());
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default visit(FiniteVectorLogicalTypeAnnotation) already falls back to visit(ListLogicalTypeAnnotation), which dispatches to the override above, so I think we can drop this:

Suggested change
@Override
public Optional<Converter> visit(
LogicalTypeAnnotation.FiniteVectorLogicalTypeAnnotation vectorLogicalType) {
return visit(LogicalTypeAnnotation.listType());
}

Type element = list3Levels.getElement();
TypeMapping child = fromParquet(element, null, element.getRepetition());
ArrowType.FixedSizeList fixedSizeList =
new ArrowType.FixedSizeList(vectorLogicalType.getNumElements());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, this maps to FixedSizeList now 👍 The other direction still loses the length: fromArrow maps ArrowType.FixedSizeList to a plain LIST (createListTypeMapping()), so FINITE_VECTOR → Arrow → Parquet gives a LIST. Should we map it back to FINITE_VECTOR(listSize) when the child is a non-nullable supported primitive, or do we keep that for a follow-up?

Comment thread parquet-hadoop/pom.xml
</dependency>
<dependency>
<groupId>org.apache.parquet</groupId>
<artifactId>parquet-encoding</artifactId>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This came in with the commit that removed the validator. Is it still needed? parquet-encoding is already on the classpath through parquet-column.

Co-authored-by: Isaac <no-reply@databricks.com>
@Fokko

Fokko commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

I reran the benchmark from my previous review on 6493ae2, using the same setup: 4M floats, uncompressed, no dictionary, single fork, 3 warmup and 5 measurement iterations (M4 Max, JDK 17).

ms/op dim V1 vector / list V2 vector / list
write 8 96.1 / 95.9 99.2 / 99.2
write 128 71.6 / 77.2 81.4 / 78.4
write 1024 72.9 / 77.4 78.2 / 78.7
readRecords 8 71.2 / 79.0 58.0 / 59.7
readRecords 128 79.0 / 67.0 60.3 / 51.5
readRecords 1024 51.2 / 49.7 39.5 / 39.8
readPages 8 3.48 / 3.40 3.36 / 3.38
readPages 128 3.42 / 3.34 3.29 / 3.33
readPages 1024 3.38 / 3.39 3.51 / 3.34

With this commit the length check runs during record assembly rather than in ParquetFileReader, so readRecords is now the comparison that matters. In the single-fork run, dim=128 looked about 17% slower for vectors, so I reran readRecords with 3 forks:

readRecords ms/op (3 forks) V1 vector / list V2 vector / list
dim 8 74.0 ± 4.3 / 72.5 ± 4.8 67.4 ± 2.6 / 65.7 ± 2.1
dim 128 68.6 ± 1.1 / 65.8 ± 0.8 57.5 ± 4.2 / 58.2 ± 3.2

That gap was a single-fork outlier. The largest difference left is about 4% at V1 dim=128, which doesn't show up for V2. So validating the lengths in the record reader doesn't add any meaningful overhead 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add VECTOR logical type reference implementation

2 participants