Repository navigation
API, Core: Handle File Type Schema Evolution - #18935
RussellSpitzer wants to merge 5 commits into
Conversation
|
CC @rambleraptor @rshdhere @dramaticlly For part 2 |
rshdhere
left a comment
There was a problem hiding this comment.
Went through this against the File Type section of the spec and poked at a few edge cases locally on the branch. The guards line up with the spec's "sub-fields cannot be added, removed, reordered, or promoted" rule, and the ID reservation behaves correctly for the cases I tried, including nesting. Things I checked that already work:
- a file inside a new struct, an existing struct, a list element, and a map value all reserve their 6 IDs, and the next added column skips the block
- delete
photothen re-addphotoin the same update gets a fresh ID (9), not the old block - case-insensitive
renameColumn("PHOTO.URI", ...)is still rejected - a non-null default on the file column itself is rejected ("must be null")
A few comments inline, mostly around test coverage and one cost question in Schema.
| } | ||
|
|
||
| private void validateFileBaseIds() { | ||
| for (NestedField field : lazyIdToField().values()) { |
There was a problem hiding this comment.
This makes every Schema build the id-to-field index eagerly in the constructor, including schemas with no file columns. Before, that index was only built here when identifier fields were set. Schemas are created often (select / projections / parsing), so is it worth gating this, e.g. only building the index when the struct actually contains a file type, or doing the check in the traversal that already happens for highestFieldId?
There was a problem hiding this comment.
This seems like the convenient way to do this, but I agree, we're sacrificing the lazy behavior. We could have a schema visitor that does the same behavior and avoid eager creation of the mapping.
There was a problem hiding this comment.
Switched to a little recursive visitor
| } | ||
|
|
||
| @Test | ||
| void addFileColumnReservesNestedIds() { |
There was a problem hiding this comment.
The description says addColumn handles a file column "at any depth", but this only covers a top-level add. I tried it locally and it works: adding list<file> then map<string, file> then a string column gives IDs 4 / 12 / 21. It'd be good to pin that with a test, since the nested path goes through assignFreshIds with the reserving NextID rather than typeForNewColumn, and that's the easier one to regress. Something like adding a struct containing a file, then another column, and asserting the IDs.
There was a problem hiding this comment.
Add tests for files in the other nested types, see Map of files and List of files below.
| } | ||
|
|
||
| @Test | ||
| void rejectsASchemaWhereAnotherColumnHoldsADerivedId() { |
There was a problem hiding this comment.
The collision is reported through the generic duplicate-key message (Multiple entries with same key: 3=photo.uri and 3=data). For a user, data having ID 3 looks perfectly valid, since the reservation is implicit. Since validateFileBaseIds already walks the file columns, could it also check that baseId + 1 .. baseId + 6 aren't taken by another field and say so explicitly (e.g. "ID 3 is reserved by file column photo")? Not blocking, just thinking about the person who hits it.
There was a problem hiding this comment.
I had this in my original implementation but I thought it wasn't worth the extra logic just to improve the message just for file types. I think you can see in the commit history the version where it does first a pass to find file types and then checks all the offset field ids.
| @Test | ||
| void acceptsAMetricsModeForAFileNestedField() { | ||
| TableMetadata metadata = | ||
| TableMetadata.newTableMetadata( | ||
| SCHEMA, | ||
| PartitionSpec.unpartitioned(), | ||
| "file:/tmp/table", | ||
| ImmutableMap.of( | ||
| TableProperties.FORMAT_VERSION, | ||
| "4", | ||
| TableProperties.METRICS_MODE_COLUMN_CONF_PREFIX + "photo.size", | ||
| "full")); | ||
|
|
||
| assertThat(metadata.properties()) | ||
| .containsEntry(TableProperties.METRICS_MODE_COLUMN_CONF_PREFIX + "photo.size", "full"); | ||
| } |
There was a problem hiding this comment.
I don't think this tests what it says. metadata.properties() just returns the map we passed in, so the only real check is that newTableMetadata doesn't throw. Can we assert the resolved mode instead? This is MetricsConfig behavior, so it probably belongs in TestMetricsConfig:
MetricsConfig config =
MetricsConfig.from(
ImmutableMap.of(TableProperties.METRICS_MODE_COLUMN_CONF_PREFIX + "photo.size", "full"),
schema,
SortOrder.unsorted());
assertThat(config.columnMode("photo.size")).isEqualTo(MetricsModes.Full.get());That would replace the containsEntry assertion here. It's also worth checking that a nested field with no override gets the default mode.
There was a problem hiding this comment.
This is dropped because of the new metrics conf path so we have expicit tests.
Previously this would have tested newTableMetadata() which would have invoked the MetricConf.validate() but point taken this was definitely not a clear expression of the test name.
| @Test | ||
| void addFileColumnReservesNestedIds() { | ||
| Schema schema = new Schema(required(1, "id", Types.LongType.get())); | ||
| Schema expected = | ||
| new Schema( | ||
| required(1, "id", Types.LongType.get()), | ||
| optional(2, "photo", Types.FileType.of(2)), | ||
| optional(9, "data", Types.StringType.get())); | ||
|
|
||
| Schema updated = | ||
| new SchemaUpdate(schema, schema.highestFieldId()) | ||
| .addColumn("photo", Types.FileType.of(0)) | ||
| .addColumn("data", Types.StringType.get()) | ||
| .apply(); | ||
|
|
||
| assertThat(updated.asStruct()).isEqualTo(expected.asStruct()); | ||
| } |
There was a problem hiding this comment.
We're only testing top-level. Should we add one at a lower depth (in map or list), since it goes through a slightly different path?
There was a problem hiding this comment.
Good call, added.
| @Override | ||
| public List<Integer> file(Types.FileType file, Iterable<List<Integer>> fieldResults) { | ||
| return struct(file.asStruct(), fieldResults); |
There was a problem hiding this comment.
Neither this nor the MetricsConfig.limitFieldIds override has a direct test, so nothing pins the order of the nested IDs or which file subfields get default metrics. TestSchemaOrder already has one test per type. Can we add a fileOrder() case there and a limitFieldIds case in TestMetricsConfig? Derive the expected IDs from Types.FileType.of(n).fields() rather than hardcoding them.
There was a problem hiding this comment.
That makes sense. I switched this around and added a property of FileType defaultInferredMetricOffsets which gets called here to assign the fields.
There was a problem hiding this comment.
Changed this again, it's now a set of fields
FILE_INFERRED_METRIC_ID_OFFSETS
| public Set<Integer> file(Types.FileType file, Iterable<Set<Integer>> fieldResults) { | ||
| return struct(file.asStruct(), fieldResults); | ||
| } |
There was a problem hiding this comment.
nit: is treating a file exactly like a struct the right default for inferred metrics? Every file column now uses six slots of the inferred-column limit, and with several file columns that could push real data columns out of default metrics. It also collects truncated bounds on inline and checksum, which I doubt help pruning. If that's intended, let's say so. Otherwise, file() could add only uri, size, and content_type by default, possibly in a later part of the series.
There was a problem hiding this comment.
Changed this around pretty dramatically. I didn't have that later in the series.
So the new version is to define the exact offsets we consider real fields (uri, size, content) and only those count towards our limit. I put the default for the other file types as "count" unless the table default is "non".
So file types now behave a little different
- Only uri, size and content are included during the "limit" phase of metric collectioN
- inline, checksum are default to count if there is still room
- Inline, checksum can be made "none" if they set none as the Table default
| @Test | ||
| void incompatibleFileAndStruct() { | ||
| Schema fileSchema = new Schema(optional(1, "photo", Types.FileType.of(1))); | ||
| Schema structSchema = | ||
| new Schema(optional(1, "photo", Types.StructType.of(Types.FileType.of(1).fields()))); | ||
|
|
||
| List<String> asFile = CheckCompatibility.readCompatibilityErrors(fileSchema, structSchema); | ||
| assertThat(asFile).hasSize(1); | ||
| assertThat(asFile.get(0)) | ||
| .as("Should complain that a struct cannot be read as a file") | ||
| .contains("cannot be read as a file"); | ||
|
|
||
| List<String> asStruct = CheckCompatibility.readCompatibilityErrors(structSchema, fileSchema); | ||
| assertThat(asStruct).hasSize(1); | ||
| assertThat(asStruct.get(0)) | ||
| .as("Should complain that a file cannot be read as a struct") | ||
| .contains("file cannot be read as a struct"); | ||
|
|
||
| assertThat(CheckCompatibility.readCompatibilityErrors(fileSchema, fileSchema)).isEmpty(); | ||
| } |
There was a problem hiding this comment.
nit: this covers three behaviors (struct as file, file as struct, file as file). Can we split it into three tests? hasSize(1) followed by get(0) also hides the actual error on failure. Prefer .singleElement().asString().endsWith("cannot be read as a file"), and drop the .as() descriptions since they just repeat the assertion.
There was a problem hiding this comment.
Behaviors are now in their own tests.
| @Test | ||
| void cannotUpdateFileNestedFieldRequirement() { | ||
| assertThatThrownBy(() -> fileUpdate().requireColumn("photo.uri")) | ||
| .isInstanceOf(IllegalArgumentException.class) | ||
| .hasMessage("Cannot change a nested field of a file column: photo.uri"); | ||
| assertThatThrownBy(() -> fileUpdate().makeColumnOptional("photo.uri")) | ||
| .isInstanceOf(IllegalArgumentException.class) | ||
| .hasMessage("Cannot change a nested field of a file column: photo.uri"); | ||
| } | ||
|
|
||
| @Test | ||
| void cannotMoveFileNestedField() { | ||
| assertThatThrownBy(() -> fileUpdate().moveFirst("photo.checksum")) | ||
| .isInstanceOf(IllegalArgumentException.class) | ||
| .hasMessage("Cannot move fields in non-struct type: file"); | ||
| assertThatThrownBy(() -> fileUpdate().moveBefore("photo.checksum", "photo.uri")) | ||
| .isInstanceOf(IllegalArgumentException.class) | ||
| .hasMessage("Cannot move fields in non-struct type: file"); | ||
| assertThatThrownBy(() -> fileUpdate().moveAfter("photo.uri", "photo.inline")) | ||
| .isInstanceOf(IllegalArgumentException.class) | ||
| .hasMessage("Cannot move fields in non-struct type: file"); | ||
| } |
There was a problem hiding this comment.
nit: these two tests each check several operations, so a failure doesn't tell you which one broke. Can we split them into one test per operation like other tests above?
| void renameFileColumn() { | ||
| Schema renamed = fileUpdate().renameColumn("photo", "image").apply(); | ||
|
|
||
| assertThat(renamed.findField("image").type()).isEqualTo(Types.FileType.of(2)); | ||
| assertThat(renamed.findField("image.uri").fieldId()).isEqualTo(3); | ||
| } |
There was a problem hiding this comment.
nit: we know the full expected schema here, so can we assert the whole struct like deleteFileColumn does? Feels like we're taking a shortcut with a half-assert.
There was a problem hiding this comment.
Yeah no point in doing that. Did the full comparison
| @Override | ||
| public List<String> file(Types.FileType readFile, Iterable<List<String>> fieldErrorLists) { | ||
| if (currentType.isFileType()) { | ||
| // the nested fields are derived from the base id, so matching ids means matching fields |
There was a problem hiding this comment.
This seems like an unnecessary (and unrelated) comment.
There was a problem hiding this comment.
I was just trying to make sure folks would understand why we aren't checking the nested fields. I'll strip it out
| } | ||
|
|
||
| private void validateFileBaseIds() { | ||
| for (NestedField field : lazyIdToField().values()) { |
There was a problem hiding this comment.
This seems like the convenient way to do this, but I agree, we're sacrificing the lazy behavior. We could have a schema visitor that does the same behavior and avoid eager creation of the mapping.
| assertThat(actual.asStruct()).isEqualTo(expected.asStruct()); | ||
| } | ||
|
|
||
| private static final Schema FILE_SCHEMA = |
There was a problem hiding this comment.
Shouldn't FILE_SCHEMA and fileUpdate() move to the top of the class with the other fields?
There was a problem hiding this comment.
A file column owns the field IDs immediately following its own, because its nested fields are derived from the enclosing ID rather than assigned. Reject a file column whose type does not derive from its field ID, and reserve the block when a file column is added or reassigned so later additions cannot collide. Nested fields of a file column cannot be renamed, deleted, promoted, moved, or otherwise updated, since the type fixes them. Generated-by: Cursor (Claude Opus 5.5)
Default inferred metrics for a file column to the uri, size, and content_type fields so a file uses three slots of the inferred-column limit instead of six. The remaining nested fields default to counts only. Validate file base IDs with a schema walk instead of forcing the ID index, and tighten and split the tests that review flagged. Generated-by: Cursor
d249fcd to
673bef6
Compare
Make the file type's nested field ID offsets public alongside NUM_NESTED_FIELDS, and keep the choice of which fields get inferred metrics private to MetricsConfig instead of on the type. Generated-by: Cursor
| public static final class FileType extends NestedType { | ||
| public static final String NAME = "file"; | ||
| public static final int NUM_NESTED_FIELDS = 6; | ||
| public static final int URI_ID_OFFSET = 1; |
There was a problem hiding this comment.
Had to make these public in order to use them in Metrics Config. Ideally I just wanted a way to define the inferred fields we cared about in Metrics config and this was the cleanest way I found to connect the two without reverse looking up fields by names.
There was a problem hiding this comment.
Sharing different perspective, since type is in API and we maintain the logic about the field to metrics mode mapping MetricsConfig as single authority, the direct name based look up in Metrics by file type does not sound too terrible. To some extend, the POSITION_DELETE_MODE today is a special case where we know the file_path and pos will have the full mode, but for metadata columns. I would say file.field(name).fieldId() for iterating uri/size/content_type subfield of FileType in limitFieldIds is not too bad.
There was a problem hiding this comment.
We would have to make the field names public then instead, or just redefine them in Metrics Conf. So we can either expose the names or the ids. So then I would have on metrics config
DEFAULT_FILE_INFERRED = {URI, ...}
Or we could just give up on the magic strings and make it
DEFAULT|_FILE_INFERRED = {"uri", ...}
There was a problem hiding this comment.
Since we provided such language in spec on which sub field to collect metrics for, and types is in API and shall not bother itself with underlying metrics, I feel the metrics config might be the best place based on how code was organized.
There was a problem hiding this comment.
I agree with that, it's how do we get those strings (or ids) from here (in types) to MetricConf
There was a problem hiding this comment.
If this is the only place require the knowledge on which field to skip stats for file type(I believe so), I would define the private list of field index instead of importing the public static constants, like in
iceberg/core/src/main/java/org/apache/iceberg/StatsUtil.java
Lines 61 to 66 in 13135ae
So we dont need to expose additional public static content in FileTypes and keep encapsulation boundary clear. While there's the downside that the index change will not be direct visible here, but we can have test to cover this and catch the problem.
There was a problem hiding this comment.
Yeah I guess that's fine :(
| assertThat(config.metricsFieldIds()).containsExactly(33, 4); | ||
| } | ||
|
|
||
| @Test |
There was a problem hiding this comment.
@danielcweeks See here and below for the new metric conf behaviors
|
@danielcweeks + @rshdhere I went over all your comments and adjusted the PR. The biggest change is the metrics conf handling behavior. Previously we were doing a bit of reverse name lookup and just treating all the fields of File like a struct. To try to match our text in the Spec a bit better and from Dan's suggestion I changed it to only count the 3 recommended fields as metric-able by default. The others are only set to "count" unless the table default is none. |
|
|
||
| @Test | ||
| void cannotPromoteFileNestedField() { | ||
| assertThatThrownBy(() -> fileUpdate().updateColumn("photo.size", Types.LongType.get())) |
There was a problem hiding this comment.
photo.size is already a long type before promotion, maybe change to StringType instead to illustrate the point?
There was a problem hiding this comment.
Sure, the name of the test is a bit confusing here. I'll switch the test name because you can't really promote any of these columns anyway. I'll just say "cannotUpdateFileNestedFieldTYpe"
| return newId; | ||
| }); | ||
|
|
||
| TypeUtil.GetID tracked = |
There was a problem hiding this comment.
nit: find a bit confusing on the purpose of tracked. I think we are basically recording the reassignment of the id by bookkeeping both in the 2 way map?
There was a problem hiding this comment.
Yeah it's because we have a few weird callers that (metadata/partition things) which will change the field ids and we need a way to map them back to their real id's so that we can apply filters to them.
We could alternatively just break if we see a file when calling this. It doesn't come up very much. Currently the only non-direct api call of this that I think matters is the Position_Deletes metadata table.
There was a problem hiding this comment.
On breaking when we see a file: I'd keep it working. PositionDeletesTable goes through this path for every table, so throwing on a file would make position_deletes unusable for any table that has a file column.
There's still one gap in the current version, though. The position_deletes GetID matches by ID value, so any row.* column whose ID equals a partition field ID (both start at 1000) gets reassigned too. A plain column just gets silently re-IDed, but a file column now throws. I reproduced it on 9ed30e4 with a wide table whose file column lands on ID 1000, partitioned by identity(id) (partition field 1000):
new PositionDeletesTable(table)
-> UnsupportedOperationException: Cannot reassign file field 1000 to 1007: this GetID does not implement get(int, Type)
The same table with a string column at 1000 loads fine. Partition fields are never file-typed, so could the GetID in PositionDeletesTable.calculateSchema override get(int, Type) and return oldId for file types? I'm happy to share the test if it helps.
There was a problem hiding this comment.
Ugh metadata tables :) I'll grab your test and patch it.
A GetID that only implements get(int) now works on schemas with file columns as long as it keeps the file field's ID, so metadata table schemas that re-ID partition fields no longer fail for tables with file columns. Also simplifies ID reassignment tracking in Schema and renames a SchemaUpdate test to describe the rejected type update. Generated-by: Cursor
MetricsConfig defines the offsets of the file fields that get inferred metrics itself, so FileType does not need to expose its ID layout. Generated-by: Cursor
| public final class MetricsConfig implements Serializable { | ||
|
|
||
| private static final Logger LOG = LoggerFactory.getLogger(MetricsConfig.class); | ||
| private static final int FILE_URI_ID_OFFSET = 1; |
| private static final Schema FILE_SCHEMA = | ||
| new Schema( | ||
| required(1, "id", Types.LongType.get()), | ||
| optional(2, "photo", Types.FileType.of(2)), |
There was a problem hiding this comment.
nit: would be great to have at least 1 test which have file type is non-top-level field, try with delete column or rename or any mutation applied to observe the expected result.
There was a problem hiding this comment.
Not a problem, I'll add
| public static final class FileType extends NestedType { | ||
| public static final String NAME = "file"; | ||
| public static final int NUM_NESTED_FIELDS = 6; | ||
| private static final int URI_ID_OFFSET = 1; |
There was a problem hiding this comment.
nit: with current logic probably dont even need changes in this class
There was a problem hiding this comment.
Yeah but less magic numbers now :) I think it's worth keeping
There was a problem hiding this comment.
I probably should have done this in the last PR
| private static final int FILE_SIZE_ID_OFFSET = 3; | ||
| private static final int FILE_CONTENT_TYPE_ID_OFFSET = 4; | ||
| private static final int[] FILE_INFERRED_METRIC_ID_OFFSETS = { | ||
| FILE_URI_ID_OFFSET, FILE_SIZE_ID_OFFSET, FILE_CONTENT_TYPE_ID_OFFSET |
There was a problem hiding this comment.
The spec says "Writers should produce statistics for uri, content_type, and inline fields; other fields may be omitted." Here the inferred fields are uri, size, and content_type, so size gets bounds while inline only gets counts. Should this follow the spec, or should the spec be updated to match?
| for (Types.NestedField field : TypeUtil.indexById(schema.asStruct()).values()) { | ||
| if (field.type().isFileType()) { | ||
| for (Types.NestedField nested : field.type().asFileType().fields()) { | ||
| builder.put(schema.findColumnName(nested.fieldId()), fileMode); |
There was a problem hiding this comment.
write.metadata.metrics.default should apply to all columns, but here the inferred file fields (like size and content_type) are always put in the map as counts. Within write.metadata.metrics.max-inferred-column-defaults (100 by default), defaultColumnConf overrides that later. Past the limit, nothing does, so they stay at counts while a normal column past the limit gets the configured default.
For example, this fails on the current code (size and content_type are counts):
@Test
void configuredDefaultAppliesToFileFieldsPastTheInferredLimit() {
Schema schema =
new Schema(
required(1, "id", Types.LongType.get()), optional(2, "photo", Types.FileType.of(2)));
Map<String, String> properties =
Map.of(
TableProperties.DEFAULT_WRITE_METRICS_MODE,
"full",
TableProperties.METRICS_MAX_INFERRED_COLUMN_DEFAULTS,
"1");
MetricsConfig config = MetricsTestUtil.from(properties, schema);
assertThat(config.columnMode("photo.size")).isEqualTo(MetricsModes.Full.get());
assertThat(config.columnMode("photo.content_type")).isEqualTo(MetricsModes.Full.get());
}Should this loop only set offset, checksum, and inline? I tried that locally and the existing tests still pass.
Part 2 of #17919, following #18252.
A file column's six nested fields take the IDs immediately after its own, so the type must stay in step with the field that holds it and those IDs must never go to another column. This PR enforces that in schema construction and evolution.
UpdateSchema.addColumnnow gives a file column, at any depth, a base ID that matches its field and reserves the derived block so later columns cannot collide. Replacing a table reserves the block when a column becomes a file. Committing file columns still needs schema JSON serialization, which comes in Part 3.API changes (no new public methods)
Schemarejects a file column whose type's base ID differs from its field ID. A column that takes a reserved ID is rejected by the existing duplicate-ID check.CheckCompatibilityaccepts reading a file as a file and reports any other type as incompatible.Core changes
UpdateSchemarejects renaming, deleting, promoting, or changing the requirement, doc, or default of a file's nested fields.MetricsConfigandSchemaOrdervisit a file like a struct. Table metadata creation reaches both visitors after Core: V4 refactor metrics config #18269 and Core: Return all metrics field IDs, in schema order #18331.Series
The following parts are draft branches in my fork. They are not open PRs and will change as this one is reviewed. Each link shows only that part's diff against the previous part.
AI Disclosure