Repository navigation
API: Add the file type - #18252
API: Add the file type#18252
Conversation
Add a `file` type for referencing a range of bytes stored inline or in an external file, as defined by the Parquet FILE logical type. The type is its own NestedType with its own TypeID rather than a struct subclass, so visitors cannot silently treat it as a struct where that would be wrong, while the default visitor behavior still falls back to struct handling for the majority of visitors that need no special casing. A file's six nested fields are derived from the ID of the field that holds the type rather than stored in the schema, which keeps the structure immutable. ID assignment reserves the derived block so that a later column cannot be given an ID that the file already owns. The type is gated to format version 4. Generated-by: Cursor
| return false; | ||
| } | ||
|
|
||
| return baseId == ((FileType) other).baseId; |
There was a problem hiding this comment.
This strict equality doesn't look quite right. Nested field does more checks and technically you could have id matches with other fields differing?
There was a problem hiding this comment.
I believe the only way for the other fields to differ is reflection. The class is final, the constructor is private, and FileType.of(baseId) is the only way to create one, so the nested fields are always derived from baseId. We could compare the subfields too, but I don't think a mismatch is possible.
There was a problem hiding this comment.
Following a suggestion above from @dramaticlly , I just dropped the "fields" entirely for a transient lazy "fieldsList" which is generated deterministically when "fields()" is called. Hopefully this makes it clearer that the only "state" that this class has is the "baseId"
| this.fields = | ||
| ImmutableList.of( | ||
| NestedField.optional(baseId + 1, URI, StringType.get()), | ||
| NestedField.optional(baseId + 2, OFFSET, LongType.get()), | ||
| NestedField.optional(baseId + 3, SIZE, LongType.get()), | ||
| NestedField.optional(baseId + 4, CONTENT_TYPE, StringType.get()), | ||
| NestedField.optional(baseId + 5, CHECKSUM, StringType.get()), | ||
| NestedField.optional(baseId + 6, INLINE, BinaryType.get())); |
There was a problem hiding this comment.
if the subfields are known and deterministic, shall we consider using the NestedField[] instead like in struct? They are anyway derived and not used in equal and hash, it shall also make kyro serializer happy
There was a problem hiding this comment.
That's a good idea. I totally forgot why we needed the lazy list there. I had removed it because I was thinking this is deterministic so why do we need a lazy list.
There was a problem hiding this comment.
I decided to do this a little differently and make @danielcweeks 's comment a bit more clear as well. We now only have a "transient" fields just like in struct type which is only set lazily and completely deterministically based on "baseId". There is no state serialized with the FileType object "except" baseId which fixes the kryo issue as well.
FileType's nested fields are fully determined by its base ID, so they are no longer stored. This makes equality on the base ID cover all state and lets Kryo round-trip the type, which failed on the stored ImmutableList. Generated-by: Cursor (Claude Opus 5.5)
| * @param assignedType the original type with IDs assigned | ||
| * @return the type the field should have | ||
| */ | ||
| static Type applyAssignedId(Type original, int newId, Type assignedType) { |
There was a problem hiding this comment.
can we rely only on visitor output assignedType?
There was a problem hiding this comment.
Yep definitely, I think this was left over from an earlier refactor I did before I stripped out the "original" usage.
| private int idFor(int id) { | ||
| return getID.get(id); | ||
| private int idFor(int id, Type type) { | ||
| return type.isFileType() ? getID.get(id, Types.FileType.NUM_NESTED_FIELDS) : getID.get(id); |
There was a problem hiding this comment.
looks like there's constant need to figure out the reserved id space of given type, 6 for file and 0 otherwise, maybe we can move it to TypeUtils so this can be simply
| return type.isFileType() ? getID.get(id, Types.FileType.NUM_NESTED_FIELDS) : getID.get(id); | |
| return getID.get(id, TypeUtil.numReservedIds(type)); |
There was a problem hiding this comment.
Good idea, let me do that.
| Types.NestedField existingField = baseField(name(currentId)); | ||
| if (canReuseBaseId(existingField, type)) { | ||
| return existingField.fieldId(); | ||
| } | ||
|
|
||
| return type.isFileType() ? nextId.get(Types.FileType.NUM_NESTED_FIELDS) : nextId.get(); |
There was a problem hiding this comment.
looks like the logic is find the filed by id and look up in the schema, then we can make type based decision on whether to reuse the base id. we can potentially inline the logic of canReuseBaseId and baseField? Something like
private int idFor(int currentId, Type type) {
String fullName = name(currentId);
if (baseSchema != null && fullName != null) {
Types.NestedField baseField = baseSchema.findField(fullName);
// a file derives its nested field IDs from its own ID, so reusing a base ID is only safe when
// the base field was already a file and had reserved that block
if (baseField != null && (!type.isFileType() || baseField.type().isFileType())) {
return baseField.fieldId();
}
}
return nextId.get(TypeUtil.numReservedIds(type));
}| Type.TypeID.GEOMETRY, 3, | ||
| Type.TypeID.GEOGRAPHY, 3); | ||
| Type.TypeID.GEOGRAPHY, 3, | ||
| Type.TypeID.FILE, 4); |
There was a problem hiding this comment.
Schema accepts a file whose base ID doesn't match the field that holds it:
new Schema(
optional(2, "f", Types.FileType.of(9)),
required(3, "x", Types.LongType.get()));This builds fine. f's sub-fields come out as 10–15, and x takes ID 3 inside the block the spec reserves for f ("reserves the root field's ID plus six consecutive IDs"). The same happens for a list element, e.g. ListType.ofOptional(2, FileType.of(20)).
#17821 rejected this in validateFileFields with Invalid file column %s: nested field IDs are derived from %s, not %s. Should that check come back? Overlaps with the derived IDs are already caught by IndexById, so only the mismatch check seems to be missing.
There was a problem hiding this comment.
Yep, I intentionally broke the original PR into what I hoped were self-contained chunks. Part 2 (of 8) handles reserving the derived ID block: it adds the base ID validation in Schema and forwards get(oldId, numReserved) through the reassignIds wrapper, with tests for both cases.
In this PR a file column can't leave the API module: SchemaParser doesn't know the type until part 3, so no table metadata can contain one yet.
| * @param numReserved number of IDs after the new ID that must not be assigned | ||
| * @return a new field ID | ||
| */ | ||
| default int get(int oldId, int numReserved) { |
There was a problem hiding this comment.
Schema.reassignIds wraps the caller's GetID in a lambda that only implements get(int). So a file column always reaches this default, even when the caller passes reassignConflictingIds, which this PR teaches to reserve IDs:
new Schema(
ImmutableList.of(optional(2, "f", Types.FileType.of(2))),
TypeUtil.reassignConflictingIds(ImmutableSet.of(2), ImmutableSet.of(1, 2, 3)));
// UnsupportedOperationException: Cannot reserve 6 IDs for field 2: reserving IDs is not supportedCalling TypeUtil.assignIds directly with the same GetID works and gives 4: f: optional file. #17821 forwarded get(oldId, numReserved) in that wrapper. The only caller today is BaseSparkScanBuilder, with partition metadata fields, so nothing breaks yet.
There was a problem hiding this comment.
Same answer as above -
Yep, I intentionally broke the original PR into what I hoped were self-contained chunks. Part 2 (of 8) handles reserving the derived ID block: it adds the base ID validation in Schema and forwards get(oldId, numReserved) through the reassignIds wrapper, with tests for both cases.
In this PR a file column can't leave the API module: SchemaParser doesn't know the type until part 3, so no table metadata can contain one yet.
Centralize the number of IDs a type reserves in TypeUtil.numReservedIds and derive the rebuilt file type from the visitor result alone, removing the duplicated file-type checks across the ID assignment visitors. Generated-by: Cursor (Claude Opus 5.5)
dramaticlly
left a comment
There was a problem hiding this comment.
LGTM, excited for the follow ups!
rambleraptor
left a comment
There was a problem hiding this comment.
I've got a (hopefully quick) question around pruning columns. Thanks for the PR! Excited to see the FILE type move forward
| * | ||
| * <p>Override this to distinguish a file from a struct. | ||
| */ | ||
| public T file(Types.FileType file, List<T> fieldResults) { |
There was a problem hiding this comment.
It looks like this has been overridden everywhere except PruneColumns, which means that PruneColumns is treating FILEs as structs.
Would we not want to project all of the columns of a File type?
There was a problem hiding this comment.
We talked about this before and the basic question is, "Should you be allowed to look at file metadata without pulling the file bytes?" If the answer is "yes" then "Inline" must be prunable. So for example, if we want
SELECT file.content_type FROM Table to be possible without reading every inlined byte, we need to be able to prune.
I'm open to other opinions but I think we want to support pruning various bits out of the File in isolation.
There was a problem hiding this comment.
I agree with pruning out specific bits. Especially if the inline bytes portion is going to be large.
| private static final String CHECKSUM = "checksum"; | ||
| private static final String INLINE = "inline"; | ||
|
|
||
| public static FileType of(int baseId) { |
There was a problem hiding this comment.
Something feels a bit conceptually off to have to publicly expose this. The nested IDs are fully determined by the holding field's ID, so callers shouldn't be supplying it imo. We can run into situations where the two disagree. For example, supporting addColumn("photo", FileType.of(?)) would mean the caller guesses an ID before one is assigned, and we'd need special-cased file handling in ID assignment to correct it.
I see part 2 adds validation for the mismatch, but having to validate it suggests the API exposes a value callers shouldn't choose.
Have we considered an ID-free FileType.get(), then binding the base ID internally when the NestedField (or list/map) is built? That way there's nothing to mismatch, and applyAssignedId goes away.Sorry if this was discussed before, is feel free to just point me to any context...
There was a problem hiding this comment.
This is part of why FileType is "kinda a nested type" and "kinda a primitive" is complicated. All of my decisions here are leaning towards "nested type." The ID here is shorthand for filling in the nested fields, the same way ListType.ofOptional(elementId, ...) and MapType.ofOptional(keyId, valueId, ...) take IDs for nested fields the caller doesn't really own.
That also covers the addColumn case: UpdateSchema already reassigns a nested type's field IDs when it's added, so addColumn("photo", FileType.of(0)) works like adding a list with a placeholder element ID. You're right that it costs some file-specific handling in ID assignment (applyAssignedId).
An ID-free FileType.get() moves the type from a "nested" form toward a "primitive" one, and IMHO that brings three odd behaviors:
-
Equality. I think there are 3 ways we could handle this but each is ... odd:
- Unbound equals every bound file. Bound types = if they match in field id. This breaks transitive equality.
- All file types are equal, like primitives: schema comparison still works because
NestedFieldcompares IDs, but anything keyed byTypebreaks. For example,TypeToSchema.WithTypeToNamecaches Avro records by type, so two file columns would share one cached record and its nested field IDs. - Compare
baseId, with unbound as its own value: this is safe, butfield.type().equals(FileType.get())is always false for a real column, which is odd for something that looks like a singleton. We could probably put in a static rule to forbid writing this code with a warning to use "isFileType()"? A little of a foot-gun, but this is what I would pick if we want to go this way.
-
fields(). An unbound file type can't return nested fields with IDs, sofields()would have to throw until the type is bound to a field. Structs, lists, and maps always have their nested fields from construction. We'd have "bound" and "unbound" file types that behave differently, which makes the type stateful in a way I'd like to avoid. -
Constructors change the type.
NestedField.of(5, "photo", FileType.get()).type()would return a different type than the one passed in. No other type is rewritten when a field is built, so that's a trap for future code.
If we're OK with those three behaviors, the workable version is the last equality option (compare baseId, unbound is distinct), and I think we can switch to it. I'd need to check the rest of the stack, but I expect that version is safe.
For what it's worth, Claude tells me "the risks are concentrated, and simplifications are wide" for what that's worth.
There was a problem hiding this comment.
For historical context I posted a few poc's a while back where we did other appraoches
FileType is a Primtive and we promote it to "nested" when we do pruning and other nestedy things
FileType is a Struct and we have special rules around schema changing and field assignment
FileType is a Special Nested Type and we have special visitors
There was a problem hiding this comment.
Posted a top level review comment, but i think i'm good with this. Only thing is could we add Javadoc on of saying the ID has to match the holding field and gets reassigned when the column is added, and keep the mismatch check in one place? Also just to double check, do the later parts add any other callers of FileType.of? I could only see #17821 and this PR.
There was a problem hiding this comment.
Added Javadoc on of. The mismatch check will be in the Schema constructor in Part 2.
The callers of FileType.of are:
- ID assignment (applyAssignedId, this PR)
- UpdateSchema when adding a column (Part 2),
- schema JSON parsing (Part 3)
- ORC schema conversion (Part 7, from the field's iceberg.id).
Also i'll just post the rest of the PR's now just incase someone wants to read ahead but i'll have to keep restacking as we change the earlier parts so i'll leave all the others as drafts till we get to them.
ReassignConflictingIds only remaps colliding IDs for read projections such as Spark's metadata columns, which never contain a file type, so it does not need to reserve derived ID blocks. Restore its original behavior; reserving through it falls back to the GetID default, which rejects file types. Generated-by: Cursor (Claude Opus 5.5)
|
@danielcweeks + @amogh-jahagirdar Could you please check this out again? I'd like to make some progress here since I have a lot of follow up PR's ready to go. Let me know if we want to continue this way. |
There was a problem hiding this comment.
Thanks for the context. I went through #17821 and #17808 to see how this actually would get used, and I think I'm good with keeping of(id).
The advantage I see is that it follows what list and map already do. The nested IDs are there when the type is built, the type is immutable, and equality compares IDs the same way, so we don't need anything new in NestedField.
My concern was callers supplying an ID they shouldn't have to choose, but looking at where FileType.of gets called in #17821, that's pretty narrow. ID assignment, schema parsing, and UpdateSchema when adding a column all already have the real field ID. A placeholder only shows up if someone hand writes FileType.of(0) for addColumn, and that gets reassigned, same as a list's element ID.
With get() callers would never pass an ID, which is nicer, but NestedField would have to swap in a bound copy of the type it was given, and no other type works that way. We'd also still have to decide what happens when an already bound type goes on a field with a different ID, so it doesn't fully get rid of the mismatch question.
Treating it as a pure primitive like variant gives us the simplest API, but we need to prune and look up its subfields by ID, and a type that can't produce those IDs on its own means passing the field ID through all the traversals. Treating it as a regular nested type doesn't work either, since users can't change the subfields, the IDs have to be a reserved block derived from the holding field, and it's written to the schema as a single file type rather than six fields. So as you said , we're inevitably in this "partially nested/partially primitive" world and we just have to optimize for clarity in terms of how we expect clients to best consume this information
Generated-by: Cursor (Claude Opus 5.5)
There was a problem hiding this comment.
Thanks @RussellSpitzer, been reviewing the tests. Overall the scenarios look great and cover a lot of the ID assignment cases! I wonder if we could structure each test around an input schema and a full expected schema, so a reader can verify it just by comparing the two. Right now most of them spot check a few IDs plus counts like highestFieldId and indexById size, so you have to work out the ID layout in your head to check them. TestTypeUtil already follows this pattern with assertThat(actual.asStruct()).isEqualTo(expected.asStruct()). Since file equality is by base ID, FileType.of(4) in the expected schema covers the nested IDs, and the reserved block shows up as a gap in the IDs. A few other things that may help:
- Using input IDs that are clearly different from the output (e.g. 100+), so it's obvious what got reassigned.
- Starting the counter at
base.highestFieldId()instead of a magic number. - Adding a new column after the file so you can see where the reserved block ends.
For example, freshIdsReserveANewBlockWhenABaseColumnBecomesAFile could just assert the assigned schema equals id=1, photo=4 (file), data=3, notes=11.
| import org.apache.iceberg.Schema; | ||
| import org.junit.jupiter.api.Test; | ||
|
|
||
| class TestFileType { |
There was a problem hiding this comment.
Ideally, we'd be able to test the behavior of file types in the existing testTypes() suite in TestTypeUtil itself, since that's where we cover every other type for project, getProjectedIds, indexById, indexNameById and reassignDoc. The ID assignment tests there assume the type stays the same when the field's ID changes, which isn't true for a file, but the other five keep it at ID 1, so FileType.of(1) should work. Could we add a second method source with testTypes() plus FileType.of(1) for those five? I think getProjectedIds in particular is worth covering, since a file goes through the struct fallback there.
There was a problem hiding this comment.
Added in a new source that has File as part of its' test list. I figure we will have other weird types int he future that may need to get grouped in with File for something like this. ... Maybe vector?
| } | ||
|
|
||
| @Test | ||
| void isNotEqualToAFileWithDifferentBaseId() { |
There was a problem hiding this comment.
Minor: could we assert FILE.hashCode() equals FileType.of(5).hashCode() instead, and drop the hashCode check in isNotEqualToAStructWithTheSameFields? (The hashCode contract requires equal objects to have the same hash, but unequal objects can collide.)
| } | ||
|
|
||
| @Test | ||
| void freshIdsReuseBaseSchemaIdsWhenTheBaseColumnIsAlsoAFile() { |
There was a problem hiding this comment.
Could we add a case with a file inside a nested struct (e.g. meta: struct<photo: file>)? With a base schema, AssignFreshIds looks up the base field by full name and ReassignIds looks it up in the matching source struct, so I think it's worth checking that both reuse and rebuild the file the same way as a top-level column.
| } | ||
|
|
||
| @Test | ||
| void freshIdsReserveANewBlockWhenABaseColumnBecomesAFile() { |
There was a problem hiding this comment.
I think this needs to assert photo's ID directly. All of the photo assertions are relative to photo.fieldId(), so if we regressed and reused the base ID 2, they'd still pass (highestFieldId would be 2 + 6 = 8). Only the indexById size check would catch it. Could we add assertThat(photo.fieldId()).isEqualTo(4)?
There was a problem hiding this comment.
I switched a lot of these checks to just check the whole file as struct so we just are comparing everything.
| } | ||
|
|
||
| @Test | ||
| void freshIdsReserveForFilesInListsAndMaps() { |
There was a problem hiding this comment.
Could we also cover a list element or map value reusing its base ID when there's a base schema? freshIdsReuseBaseSchemaIdsWhenTheBaseColumnIsAlsoAFile only covers a top-level column, but for photos.element the base field is found by its full name, so I think it's worth checking that it reuses the base ID and gets rebuilt from it the same way.
Pass the field type to NextID and GetID so the interfaces decide how many IDs a type reserves, make SchemaVisitor.file() throw by default like variant(), and restructure the file type ID tests around full expected schemas. Generated-by: Cursor (Claude Opus 5.5)
| Arguments.of(Types.GeographyType.of("srid:4269", EdgeAlgorithm.KARNEY))); | ||
| } | ||
|
|
||
| private static Stream<Arguments> testTypesWithFile() { |
There was a problem hiding this comment.
We can rename this when we get another weird type
Generated-by: Cursor (Claude Opus 5.5)
amogh-jahagirdar
left a comment
There was a problem hiding this comment.
Thanks @RussellSpitzer !
|
Merged, Thanks for the reviews @dramaticlly , @danielcweeks and @amogh-jahagirdar . I'll raise the next pr soon |
|
Also thanks to @rambleraptor and @rshdhere! Sorry I forgot to add you in my message above |
Initial Implementation of #17918 - Part of #17919 - Built from work on #17821 after review and iteration. This is one of a series of PRs for full File Support. This PR covers just the API module changes and initial visitor changes.
Add a
filetype for referencing a range of bytes stored inline or in an external file, as defined by the Parquet FILE logical type. The type is its own NestedType with its own TypeID rather than a struct subclass, so visitors cannot silently treat it as a struct where that would be wrong, while the default visitor behavior still falls back to struct handling for the majority of visitors that need no special casing.A file's six nested fields are derived from the ID of the field that holds the type rather than stored in the schema, which keeps the structure immutable. ID assignment reserves the derived block so that a later column cannot be given an ID that the file already owns.
The type is gated to format version 4.
Generated-by: Cursor
Follow-up parts (preview only)
These are draft branches in my fork, linked as a preview of where the series goes. They are not open PRs and will change as this PR is reviewed. Each link shows only that part's diff against the previous part.