diff --git a/changelog.d/ITL-1000.md b/changelog.d/ITL-1000.md new file mode 100644 index 000000000..8af148acd --- /dev/null +++ b/changelog.d/ITL-1000.md @@ -0,0 +1,3 @@ +### Breaking changes and migration + +- The `content_hash` of a `DerivedSource` built with `tags_as_data` now includes the mapping (ITL-1000). Before, two mappings that gave the same schema (two tag columns of one type copied to one name) and shared an explicit `source_id` had the same `content_hash` although the copy held different values; the sorted mapping items are now part of the identity. The `content_hash` moves whenever `tags_as_data` is given, with the default `source_id` as well as an explicit one, because the mapping is appended in both cases. The `source_id` (the default id was already built from the mapping), record IDs, source-info tokens and `pipeline_hash` do not move, and a `DerivedSource` without `tags_as_data` (a job node's stored records) is unchanged. `tags_as_data` was added in 0.3 and has not been released, so no released store holds a hash that moves. A store written by an unreleased 0.3 build with a `tags_as_data` `DerivedSource` gives that source, and the nodes downstream of it, a new node identity: those pipeline records are recomputed, and the old rows stay in place. The golden values `schema:derived_source_tags_as_data.{default_source_id,named_source_id,record_id_column}.content_hash` changed with it. diff --git a/design_issues/ITL-1000-DS1.md b/design_issues/ITL-1000-DS1.md new file mode 100644 index 000000000..e1cd42cf4 --- /dev/null +++ b/design_issues/ITL-1000-DS1.md @@ -0,0 +1,10 @@ +## `src/orcapod/core/sources/derived_source.py` — identity of a `tags_as_data` source (ITL-1000) + +### DS1 — `DerivedSource` `tags_as_data` mapping was not part of the content hash +**Status:** resolved +**Severity:** medium +**Issue:** ITL-1000 + +A `DerivedSource` with `tags_as_data` hashed `(class name, output schema, source_id)`. Two mappings that give the same schema (two tag columns of one type copied to one name) with one explicit `source_id` shared a content hash, although the copy holds different values. + +**Fix:** `DerivedSource.identity_structure` appends `("tags_as_data", sorted items)` when the option is given, as `SpiralDBTableSource` does (ITL-996). The `pipeline_hash` stays schema-only; the plain form is unchanged. Developer decision, November 12, 2026. diff --git a/docs/concepts/sources.md b/docs/concepts/sources.md index 52c596b5e..d96e54b1d 100644 --- a/docs/concepts/sources.md +++ b/docs/concepts/sources.md @@ -394,7 +394,8 @@ before. Things to know: has provenance columns of its own, like the result of handing the same rows to `ArrowTableSource`. Use it in place of the origin. Its default `source_id` is `tags_as_data:` followed by a labelled hash built from the origin's identity and the mapping, so - it changes when either changes; pass `source_id=` to choose a name. + it changes when either changes; pass `source_id=` to choose a name. The mapping is part of the + source's `content_hash` either way, so two mappings never share one. - A saved job reloads a derived source whenever its origin can be reloaded (a CSV file, a Delta table, a database table). Over an in-memory source it is saved but not reloaded, like the in-memory source itself. diff --git a/docs/reference/hashing.md b/docs/reference/hashing.md index 8fbb7a3d3..53c417e99 100644 --- a/docs/reference/hashing.md +++ b/docs/reference/hashing.md @@ -322,7 +322,7 @@ The table below shows what each class returns from `identity_structure()`. | `ArrowTableStream` | `(producer, argument_symmetry(upstreams))` | Falls back to `(class_name, as_table(all_info=True), tag_columns)` when no producer or producer is not a `PipelineElementProtocol` | | `RootSource` (`ArrowTableSource`) | `(class_name, (tag_schema, data_schema), source_id)` | `source_id` is the explicit user-supplied value; when not provided it defaults to `::` of the Arrow hash of the raw table (site 4). Data-inclusive base case of the Merkle chain. | | `DerivedSource` | Origin node's `content_hash` | Inherits its generating node's identity | -| `DerivedSource` with `tags_as_data` | `(class_name, output_schema, source_id)`; the default `source_id` is `tags_as_data: