Repository navigation
GH-3877: Add FINITE_VECTOR logical type support - #3876
divjotarora wants to merge 19 commits into
Conversation
Fokko
left a comment
There was a problem hiding this comment.
The biggest issue is around the validation. It feels to me that a lot of visitor that are meant to do other things, are now also validating if it is a valid FINITE_VECTOR. I left some other comments as well. Let me know what you think
There was a problem hiding this comment.
Eventually, these should go in https://github.com/apache/parquet-testing/
There was a problem hiding this comment.
Yes, that's definitely the goal here
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Fokko
left a comment
There was a problem hiding this comment.
Thanks @divjotarora for iterating on this, I think we're getting close 👍 Now, my main concern is the performance penalty that we're having because of the validation.
|
Another thing, I think we don't need to collect that much statistics for the vector:
We can also do this in a separate PR |
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Fokko
left a comment
There was a problem hiding this comment.
Thanks for iterating on this! I ran a benchmark comparing FINITE_VECTOR(n) against a plain LIST holding the same data (4M floats, uncompressed, no dictionary, float[] in/out through a minimal WriteSupport/ReadSupport so the object model doesn't dominate). readPages reads row groups and pages without record assembly, which is where the count checks run:
| ms/op | dim | V1 vector / list | V2 vector / list |
|---|---|---|---|
| write | 8 | 94.9 / 104.0 | 90.5 / 98.6 |
| write | 128 | 75.4 / 76.4 | 78.2 / 82.4 |
| write | 1024 | 70.5 / 71.2 | 71.9 / 82.0 |
| readRecords | 8 | 74.9 / 71.8 | 50.7 / 60.4 |
| readRecords | 128 | 66.3 / 67.7 | 49.3 / 49.3 |
| readRecords | 1024 | 43.6 / 44.5 | 37.8 / 37.1 |
| readPages | 8 | 3.55 / 3.42 | 3.39 / 3.38 |
| readPages | 128 | 3.37 / 3.33 | 3.29 / 3.51 |
| readPages | 1024 | 3.44 / 3.30 | 3.43 / 3.23 |
There's no measurable difference in either direction (single fork, so the swings both ways are run-to-run noise), so removing the per-value validation took care of my performance concern 👍 I can open the benchmark as a separate PR.
My main remaining point is about the reader-side count checks (see inline): now that we delegate validation to the engine, they only catch some invalid files. The rest are small things.
| VectorColumn vector = vectorColumns.get(column.getPath()); | ||
| if (vector == null) return; | ||
| org.apache.parquet.column.statistics.Statistics<?> stats = column.getStatistics(); | ||
| long nulls = stats != null && stats.isNumNullsSet() ? stats.getNumNulls() : -1; |
There was a problem hiding this comment.
This turns null_count from the statistics into a hard read failure. Statistics are optional, and we've had writers producing incorrect ones before, so a file with valid data but a bad null_count becomes unreadable. One more reason to drop the count checks (see above).
| } | ||
| } | ||
|
|
||
| public static class FiniteVectorLogicalTypeAnnotation extends ListLogicalTypeAnnotation { |
There was a problem hiding this comment.
The spec says every non-null vector MUST have exactly num_elements elements and every element MUST be finite. Since parquet-java doesn't enforce either (testVectorValuesAreNotValidated writes NaN/Inf), could we add a Javadoc here saying so? Then users know it's up to them to write valid vectors.
| @Override | ||
| public boolean equals(Object obj) { | ||
| return obj instanceof ListLogicalTypeAnnotation; | ||
| return obj instanceof ListLogicalTypeAnnotation && !(obj instanceof FiniteVectorLogicalTypeAnnotation); |
There was a problem hiding this comment.
Nit: this way the parent needs to know about its subclass. How about:
| return obj instanceof ListLogicalTypeAnnotation && !(obj instanceof FiniteVectorLogicalTypeAnnotation); | |
| return obj != null && obj.getClass() == ListLogicalTypeAnnotation.class; |
|
|
||
| @Override | ||
| public Optional<Converter> visit( | ||
| LogicalTypeAnnotation.FiniteVectorLogicalTypeAnnotation vectorLogicalType) { | ||
| return visit(LogicalTypeAnnotation.listType()); | ||
| } |
There was a problem hiding this comment.
The default visit(FiniteVectorLogicalTypeAnnotation) already falls back to visit(ListLogicalTypeAnnotation), which dispatches to the override above, so I think we can drop this:
| @Override | |
| public Optional<Converter> visit( | |
| LogicalTypeAnnotation.FiniteVectorLogicalTypeAnnotation vectorLogicalType) { | |
| return visit(LogicalTypeAnnotation.listType()); | |
| } |
| Type element = list3Levels.getElement(); | ||
| TypeMapping child = fromParquet(element, null, element.getRepetition()); | ||
| ArrowType.FixedSizeList fixedSizeList = | ||
| new ArrowType.FixedSizeList(vectorLogicalType.getNumElements()); |
There was a problem hiding this comment.
Nice, this maps to FixedSizeList now 👍 The other direction still loses the length: fromArrow maps ArrowType.FixedSizeList to a plain LIST (createListTypeMapping()), so FINITE_VECTOR → Arrow → Parquet gives a LIST. Should we map it back to FINITE_VECTOR(listSize) when the child is a non-nullable supported primitive, or do we keep that for a follow-up?
| </dependency> | ||
| <dependency> | ||
| <groupId>org.apache.parquet</groupId> | ||
| <artifactId>parquet-encoding</artifactId> |
There was a problem hiding this comment.
This came in with the commit that removed the validator. Is it still needed? parquet-encoding is already on the classpath through parquet-column.
Co-authored-by: Isaac <no-reply@databricks.com>
|
I reran the benchmark from my previous review on 6493ae2, using the same setup: 4M floats, uncompressed, no dictionary, single fork, 3 warmup and 5 measurement iterations (M4 Max, JDK 17).
With this commit the length check runs during record assembly rather than in
That gap was a single-fork outlier. The largest difference left is about 4% at V1 dim=128, which doesn't show up for V2. So validating the lengths in the record reader doesn't add any meaningful overhead 👍 |
Rationale for this change
The FINITE_VECTOR proposal defines fixed-size vectors of non-null, finite elements on canonical LIST storage, so applications can express vector semantics while retaining existing Parquet storage and interoperability. This change brings parquet-java support in line with the proposal.
What changes are included in this PR?
Adds FINITE_VECTOR schema annotations and mandatory write validation, validates stored levels and elements during reads, and preserves LIST interoperability across existing model readers.
Are these changes tested?
The affected Maven suites for FINITE_VECTOR schemas, writing, reading, adapters, and filtered reads passed, and Spotless passed for all touched modules.
Closes #3877