Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
114 commits
Select commit Hold shift + click to select a range
0886ffd
docs(plan): ITL-910 adopt starfix 0.5 investigation and plan (WIP)
Nov 11, 2026
1b57f9b
docs(plan): ITL-910 suite findings under starfix 0.5 (WIP)
Nov 11, 2026
b9f4465
test(hashing): failing tests for starfix hash format 2 under arrow_v0…
Nov 11, 2026
ca5d8cd
feat(hashing): adopt starfix 0.5 (hash format 2) under label arrow_v0…
Nov 11, 2026
bd252f2
test(golden): regenerate golden hashes for starfix hash format 2 / ar…
Nov 11, 2026
a5ceb0a
test: re-pin format-1 digests and record-id order in starfix format 2…
Nov 11, 2026
f5433ff
docs: starfix hash format 2 / arrow_v0.2 in CHANGELOG, hashing refere…
Nov 11, 2026
bf8784f
style: ruff format and type the format-2 tests (ITL-910)
Nov 11, 2026
6dfb456
docs(plan): ITL-910 D1 design (explicit failure on foreign hash labels)
Nov 11, 2026
722a098
test(pipeline): failing tests: stores with rows under another Arrow h…
Nov 11, 2026
0c05cbd
feat(store): refuse stored rows hashed under another Arrow hash label…
Nov 12, 2026
9bef94e
fix(store): hash-label check ignores non-digest record ids, works on …
Nov 12, 2026
6df680c
docs: v0.3 refuses stores written by v0.2 (StoreHashLabelError); corr…
Nov 12, 2026
a49ba10
style: type the stored-hash-label helpers (ITL-910)
Nov 12, 2026
d5d564e
refactor: break the stored-hash-label import cycle; type the new test…
Nov 12, 2026
5363bfb
docs(plan): ITL-910 worker report
Nov 12, 2026
4f907c4
docs(plan): ITL-942 step 1 analysis of the wrong-hash findings
Nov 12, 2026
5acdf37
test(hashing): failing regression tests for the ITL-942 wrong-hash fi…
Nov 12, 2026
34960c8
feat(store): ignore rows hashed under another Arrow label instead of …
Nov 12, 2026
66517c5
docs: v0.3 ignores rows hashed under arrow_v0.1, recomputes and keeps…
Nov 12, 2026
92ffb2e
style: type the stored-hash-label filter and its tests (ITL-910)
Nov 12, 2026
bcd905b
feat(store): warn with the number of skipped rows; side-effect logs u…
Nov 12, 2026
b88f7eb
docs(plan): ITL-910 follow-up status (rows of another label ignored, …
Nov 12, 2026
69f4b98
Merge remote-tracking branch 'origin/eywalker/itl-910-adopt-starfix-0…
Nov 12, 2026
c30f04c
docs(plan): ITL-942 finding R (fixed by #292) and G re-check
Nov 12, 2026
007ea85
docs(plan): ITL-935 logical type identifiers, step 1 plan
Nov 12, 2026
7bfcb54
docs(plan): ITL-944 step 1 plan for data context v1
Nov 12, 2026
26dcbc5
feat(contexts): data context std:v1:default with a frozen-manifest ch…
Nov 12, 2026
87df847
wip(store): __data_context column, new table layouts, read filter, ol…
Nov 12, 2026
02db3f0
wip(store): function version column, tests moved to the new layouts (…
Nov 12, 2026
5c34afb
docs(plan): ITL-945 step 1 plan for the hash label rule
Nov 12, 2026
4030f55
docs(plan): ITL-945 tracer results and open points
Nov 12, 2026
9b1c2be
feat(store): __data_context on every stored row, new table layouts, o…
Nov 12, 2026
2dc0d24
test(hashing): failing tests for the hash label rule (ITL-945)
Nov 12, 2026
8a1f5a1
refactor(hashing)!: rename hash labels to <algorithm>_v1 (ITL-945)
Nov 12, 2026
db94f0f
feat(hashing)!: keep hash labels out of every hash input (ITL-945)
Nov 12, 2026
2e619aa
feat(contexts): manifest fields of the versioning design; logical typ…
Nov 12, 2026
2c4b1e3
feat(core)!: bare hex digests inside names; stream identity without t…
Nov 12, 2026
a7691dc
feat(store): function pods that take the invocation context fail on a…
Nov 12, 2026
4b9e825
feat(store)!: labelled row keys for operator-cache and cached-source …
Nov 12, 2026
de11a36
test(golden): pinned hash values after the hash label rule (ITL-945)
Nov 12, 2026
bc012b3
refactor(hashing): define each hash label once, beside its algorithm …
Nov 12, 2026
07ccf30
refactor(store)!: remove the v0.1 manifest and the v0 to v1 migration…
Nov 12, 2026
50d453e
docs: the hash label rule in the hashing reference, CHANGELOG, DESIGN…
Nov 12, 2026
18a110b
feat(store): refuse an input of another data context (ITL-944)
Nov 12, 2026
f7a7c87
fix(hashing): sort set items by canonical JSON, not str() (ITL-942 L1)
Nov 12, 2026
45787de
fix(sources): record-id system tag is large_binary everywhere (ITL-94…
Nov 12, 2026
58198d8
fix(function_pod): default function version is v0.0 for every entry p…
Nov 12, 2026
ed81f34
fix(hashing): Arrow hashed type depends on the declared type only (IT…
Nov 12, 2026
ca6fb1c
fix(operators): fold system-tag source ids over the canonical member …
Nov 12, 2026
8b05071
fix(store): address CodeRabbit review of PR #313 (ITL-944)
Nov 12, 2026
53577ff
fix(operators): column selection identity is the set of names (ITL-94…
Nov 12, 2026
55ba591
fix(streams): keep tag sorting in the producer-less stream identity; …
Nov 12, 2026
d54020d
fix(hashing): dict keys keep their type in structure hashes (ITL-942 …
Nov 12, 2026
a690970
test(hashing): pin MergeJoin input identity as intended; document the…
Nov 12, 2026
f3157b0
fix(hashing): only the hasher and type handlers write '__type__' obje…
Nov 12, 2026
a6adec1
docs: a v0.2 side-effect log is not detected at the v0.3 path (ITL-94…
Nov 12, 2026
088502a
test(hashing): regenerate golden values for the ITL-942 fixes
Nov 12, 2026
749778d
docs: CHANGELOG and DESIGN_ISSUES entries for the ITL-942 wrong-hash …
Nov 12, 2026
603e2c3
feat(logical_types): identifiers for built-in types; whole identifier…
Nov 12, 2026
a63896b
style: type the Encoded expansion and the side-effect version test (I…
Nov 12, 2026
4c0a820
docs: logical type identifiers page, changelog, design issue, agent r…
Nov 12, 2026
7b6626b
docs(plan): ITL-954 plan and the store fixture written by Orcapod v0.2.0
Nov 12, 2026
0111b39
feat(databases): list_table_paths lists the tables under a path prefi…
Nov 12, 2026
7af5cd7
docs(changelog): a fresh store does not prevent repeated side effects…
Nov 12, 2026
03b4be2
test(store): pre-v0.3 tables on the real v0.2.0 store, as written and…
Nov 12, 2026
531ab89
fix(hashing): a namedtuple's field order is part of its hash (ITL-942 T)
Nov 12, 2026
930e94b
feat(store): find tables written before v0.3 under a node's name, wha…
Nov 12, 2026
81ccd84
refactor(operators): delete the dead duplicate SelectDataColumns in f…
Nov 12, 2026
654ae9f
docs: record ITL-942 T in CHANGELOG and DESIGN_ISSUES
Nov 12, 2026
40a7102
fix(sources): default derived source ids keep the hash label (ITL-945)
Nov 12, 2026
5bf7935
docs(store): pre-v0.3 tables are found by name; list_table_paths (ITL…
Nov 12, 2026
43f88cc
test(hashing): address CodeRabbit round 1 on PR #315 (ITL-942)
Nov 12, 2026
6addbf0
fix(job): a node without a name path claims no table in the job total…
Nov 12, 2026
b8990fa
feat(logical_types): wrapper identifiers in the schema hash; golden r…
Nov 12, 2026
3a6a075
feat(logical_types): native containers carry no version; fix []/[None…
Nov 12, 2026
ed2653e
docs(plan): ITL-935 decisions and outcome
Nov 12, 2026
a853811
docs(plan): ITL-942 worker report
Nov 12, 2026
8b291c9
docs(changelog): move the ITL-954 entry to a changelog fragment (ITL-…
Nov 12, 2026
c4490d5
docs(changelog): move the ITL-910 entry to a changelog fragment (ITL-…
Nov 12, 2026
a0b3825
Merge branch 'eywalker/itl-910-adopt-starfix-05-hash-format-2-in-orca…
Nov 12, 2026
13111d5
Merge branch 'eywalker/itl-910-adopt-starfix-05-hash-format-2-in-orca…
Nov 12, 2026
2d2c02f
Merge branch 'eywalker/itl-910-adopt-starfix-05-hash-format-2-in-orca…
Nov 12, 2026
3192a8a
docs(changelog): move the ITL-945 entry to a changelog fragment (ITL-…
Nov 12, 2026
6fb74c2
docs(changelog): move the ITL-942 entry to a changelog fragment (ITL-…
Nov 12, 2026
2369d75
docs(changelog): move the ITL-935 entry to a changelog fragment (ITL-…
Nov 12, 2026
81da973
docs(changelog): move the ITL-944 entry to a changelog fragment (ITL-…
Nov 12, 2026
c19cb9f
docs(plans): ITL-967 combined hash-step merge plan
Nov 12, 2026
f2c1fd8
Merge branch 'eywalker/itl-910-adopt-starfix-05-hash-format-2-in-orca…
Nov 12, 2026
07e9bd3
Merge branch 'eywalker/itl-944-data-context-v1-frozen-manifest' (#313…
Nov 12, 2026
c1949b2
Merge remote-tracking branch 'origin/eywalker/itl-954-find-tables-wri…
Nov 12, 2026
c8f1194
Merge branch 'eywalker/itl-945-apply-the-hash-label-rule-one-format-l…
Nov 12, 2026
128b2c5
Merge branch 'eywalker/itl-942-fix-the-wrong-hash-findings-of-the-has…
Nov 12, 2026
6ba6cc3
Merge remote-tracking branch 'origin/eywalker/itl-935-design-make-the…
Nov 12, 2026
5286f7f
test(sources): failing tests and plan for default source ids (ITL-957)
Nov 12, 2026
b30b6fa
fix(sources): name every column the Arrow hasher refuses, as ValueErr…
Nov 12, 2026
3077859
feat(sources)!: default source ids built from a hash are <kind>:<labe…
Nov 12, 2026
827f4e3
docs(sources): default source id form, changelog fragment, design iss…
Nov 12, 2026
bc5eb91
Merge remote-tracking branch 'origin/eywalker/itl-967-v03-hash-step-c…
Nov 12, 2026
b14a56c
fix(review): CodeRabbit findings on the ITL-910 code, on the combined…
Nov 12, 2026
8c2066a
test(hashing): regenerate the golden hashes once for the v0.3 hash st…
Nov 12, 2026
d7dacbf
Merge remote-tracking branch 'origin/eywalker/itl-967-v03-hash-step-c…
Nov 12, 2026
fdc6362
test(pipeline): type the values read in the combined-branch tests (IT…
Nov 12, 2026
4481111
test(golden): default source ids: 8 entries move, 10 added (ITL-957)
Nov 12, 2026
b7b81b2
docs(plans): ITL-957 progress notes
Nov 12, 2026
dad1111
docs(plans): ITL-967 list of golden entries that move relative to main
Nov 12, 2026
970ccf0
Merge remote-tracking branch 'origin/main' (#316, ITL-955) into ITL-967
Nov 12, 2026
444bd35
test(hashing): regenerate the golden hashes after merging main (ITL-9…
Nov 12, 2026
c4b1e4a
docs(design-issues): move the ITL-910, ITL-944, ITL-954, ITL-945, ITL…
Nov 12, 2026
b7f7162
Merge remote-tracking branch 'origin/eywalker/itl-967-v03-hash-step-c…
Nov 12, 2026
99796a2
chore: merge main (hash step #324, squashed) into ITL-957
Nov 12, 2026
ff415da
chore: merge main (user-class kinds, #325) into itl-957
Nov 12, 2026
71faa29
test(sources): plain as_source() id is the DerivedSource default (ITL…
Nov 12, 2026
353eb44
chore: merge main (function signature hash, #323) into itl-957
Nov 12, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions .zed/rules
Original file line number Diff line number Diff line change
Expand Up @@ -509,8 +509,8 @@ Every pipeline element has two parallel hashes:
necessarily going to reflect the very data content"). A root source hashes
(class name, output schema, source_id); given a source_id, changed, added or reordered
rows leave it unchanged. Row content enters only where a source puts it into its own
identity: the default source_id of the in-memory sources (a hash of the table, used when
none is given) and the elements of a ListSource. A downstream stream or node hashes its
identity: the default source_id of the in-memory sources
(<source kind>:<algorithm label>:<hex> of the table hash, used when none is given) and the elements of a ListSource. A downstream stream or node hashes its
pod's identity plus its upstreams' content hashes; a function pod's identity is its uri,
not its body. Only a datagram (one row's tags or data) has a content_hash() that is a
digest of its values; that row hash, not the node's content hash, is what results are
Expand Down Expand Up @@ -674,6 +674,15 @@ data. The rules:
### Important implementation details

- ArrowTableSource raises ValueError if any tag_columns are not in the table.
- A source id is the source's name (developer, November 11, 2026; ITL-957). A default id
built from a hash is <source kind>:<algorithm label>:<hex> and is built only by
core/sources/source_ids.hash_source_id: never format one by hand, and never write a label
as a literal. The kind is the class's SOURCE_KIND (its config type key: arrow_table,
dict, list, data_frame, derived), or names the operation that derives the source
(tags_as_data). node.as_source() gives no id of its own; it takes the DerivedSource
default. The origin's source id is never written into a default id, and a source_id given
by the user is never changed. SourceStreamBuilder.build has no bare-hash default: it
needs a source_id or a source_kind.
- ArrowTableStream requires at least one data column; raises ValueError otherwise.
- FunctionNode Phase 1 returns ALL records in the shared pipeline_path DB table.
Phase 2 skips inputs whose hash is already in the DB.
Expand Down
13 changes: 11 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -529,8 +529,8 @@ Every pipeline element has two parallel hashes:
necessarily going to reflect the very data content"). A root source hashes
`(class name, output schema, source_id)`; given a `source_id`, changed, added or reordered
rows leave it unchanged. Row content enters only where a source puts it into its own
identity: the default `source_id` of the in-memory sources (a hash of the table, used when
none is given) and the elements of a `ListSource`. A downstream stream or node hashes its
identity: the default `source_id` of the in-memory sources
(`<source kind>:<algorithm label>:<hex>` of the table hash, used when none is given) and the elements of a `ListSource`. A downstream stream or node hashes its
pod's identity plus its upstreams' content hashes; a function pod's identity is its uri,
not its body. Only a datagram (one row's tags or data) has a `content_hash()` that is a
digest of its values; that row hash, not the node's content hash, is what results are
Expand Down Expand Up @@ -716,6 +716,15 @@ data. The rules:
### Important implementation details

- `ArrowTableSource.__init__` raises `ValueError` if any `tag_columns` are not in the table.
- A source id is the source's name (developer, November 11, 2026; ITL-957). A default id
built from a hash is `<source kind>:<algorithm label>:<hex>` and is built only by
`core/sources/source_ids.hash_source_id`: never format one by hand, and never write a label
as a literal. The kind is the class's `SOURCE_KIND` (its config type key: `arrow_table`,
`dict`, `list`, `data_frame`, `derived`), or names the operation that derives the source
(`tags_as_data`). `node.as_source()` gives no id of its own; it takes the `DerivedSource`
default. The origin's source id is never written into a default id, and a `source_id` given
by the user is never changed. `SourceStreamBuilder.build` has no bare-hash default: it
needs a `source_id` or a `source_kind`.
- `ArrowTableStream` requires at least one data column; raises `ValueError` otherwise.
- `FunctionNode.iter_data()` Phase 1 returns ALL records in the shared `pipeline_path`
DB table (not filtered to current inputs). Phase 2 skips inputs whose hash is already
Expand Down
42 changes: 42 additions & 0 deletions changelog.d/ITL-957.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
### Breaking changes and migration

#### Default source ids built from a hash say what kind of source they name (ITL-957)

A source id is the source's name, so it should make the source easy to identify (developer,
November 11, 2026). A default id built from a hash now has the form
`<source kind>:<algorithm label>:<hex>`. The one string is used unchanged as the `source_id`
system tag value and at the start of every `_source_<column>` value.

- **In-memory sources with no `source_id`.** `ArrowTableSource`, `DictSource`, `ListSource`
and `DataFrameSource` were named by the bare hex of the table hash (`000002fe…`). They are
now named `arrow_table:arrow_v1:<hex>`, `dict:arrow_v1:<hex>`, `list:arrow_v1:<hex>` and
`data_frame:arrow_v1:<hex>`. The kind is the class's config type key and the label is the
one the hash carries. `hashing.path_n_char`, when set, still shortens the hex, and only
the hex.
- **`node.as_source()`.** `FunctionJobNode.as_source()` and `OperatorJobNode.as_source()` no
longer build `<node path>:<label>:<hex>`. They take the `DerivedSource` default,
`derived:semantic_v1:<hex>` of the node's content hash, the same id as
`DerivedSource(node)`.
- **What moves** for an in-memory source with no `source_id`: the `source_id`, the values of
its `_tag_source_id::…` column, its record ids, its `_source_<column>` values, the name of
the source in a job, and the content hash of the source (not for `ListSource`, whose
content hash does not include the source id) and of everything downstream of it. No table
path holds a source id; paths that hold a content hash (a `table_scope="content_hash"`
node, the default path of a cached source) move with that hash. Pipeline hashes, schema
hashes, column names, data hashes and the paths of function tables do not move, and a
function's stored result for the same input data is still found.
- **What moves for `as_source()`:** the `source_id` and the name of the source in a job, and
nothing else: its rows are the node's stored records and its content hash is the node's.
- **Not changed:** a `source_id` you give is used exactly as given. Sources whose default is
not a hash (CSV, Delta, database tables, SpiralDB, cached and polling sources) keep their
defaults.
- **Migration:** to keep a name stable across releases, pass an explicit `source_id`. As
rule 5a of the hash label rule states, a default id holds the hash's label, so a later
rename of a label would move default ids and what derives from them.

### Changed

- Each in-memory source class and `DerivedSource` declares its kind once as `SOURCE_KIND`
(`arrow_table`, `dict`, `list`, `data_frame`, `derived`), used for the default source id,
for `to_config()["source_type"]` and for the source registry.
`orcapod.core.sources.source_ids.hash_source_id` is the single definition of the form.
32 changes: 32 additions & 0 deletions design_issues/ITL-957-S2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
## `src/orcapod/core/sources/`

### S2 — Default source ids built from a hash did not say what they name
**Status:** resolved
**Severity:** medium
**Issue:** ITL-957

A source id is the source's name: it is stored in every `_tag_source_id::…` value and at the
start of every `_source_<column>` value, and it names the source in a job. When no
`source_id` was given, the in-memory sources (`ArrowTableSource`, `DictSource`, `ListSource`,
`DataFrameSource`) used the bare hex of the Arrow table hash, `000002fe…`: 70 characters that
say neither what kind of source they name nor which algorithm produced them.
`FunctionJobNode.as_source()` and `OperatorJobNode.as_source()` built a third form of their
own, `<node path>:<label>:<hex>`, beside the `DerivedSource` default `derived:<label>:<hex>`.
The config type key of each class (`"dict"`, `"arrow_table"`, …) was written as a literal in
`to_config` and again in the source registry.

The developer decided on November 11, 2026: "if they are going to use some kind of hash value
as their source ID, it ought to have label/prefix to help identify what source type/category
it's associated with."

**Fix:** A default id built from a hash is `<source kind>:<algorithm label>:<hex>`, built only
by `core/sources/source_ids.hash_source_id`, which takes the label from the hash and raises
if the kind or the label is empty or holds a `:` (so the id never contains the `::` of a
source-info value). Each class declares `SOURCE_KIND` once; `to_config`, the source registry
and the default id use it. `SourceStreamBuilder.build` has no bare-hash default any more: it
needs a `source_id` or a `source_kind`. `as_source()` passes no id and takes the
`DerivedSource` default. `hashing.path_n_char` shortens the hex only. A given `source_id` is
never changed.

Left as they are: `PollingSource` (default `str(impl.identity())`, not a hash) and the
sources whose default is a path or a table name.
6 changes: 4 additions & 2 deletions docs/concepts/sources.md
Original file line number Diff line number Diff line change
Expand Up @@ -326,8 +326,10 @@ data, no result is ever paired with the wrong row; a function's stored result fo
is reused rather than recomputed.

Sources that accept `record_id_column` use that column's value instead, so the record ID
follows the row. Both halves of the record ID must be stable: when you omit `source_id` it
defaults to a hash of the table, which changes when the rows are reordered. Whenever row
follows the row. Both halves of the record ID must be stable: when you omit `source_id` an
in-memory source is named after a hash of its table, as
`<source kind>:<algorithm label>:<hex>` (for example `dict:arrow_v1:000002fe…`), which changes
when the rows are reordered. Whenever row
order can vary between runs, pass an explicit, stable `source_id` **and** set
`record_id_column` to a column that is unique per row; this avoids both the duplicate entries
and the repeated deliveries.
Expand Down
16 changes: 8 additions & 8 deletions docs/reference/hashing.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ appended, because it is a variant of the same system tag mechanism.)
| 1 | `content_hash()` | `SemanticAwarePythonHasher` → JSON + SHA-256 | `ContentHash` | Unique per semantic content of the object; data-inclusive |
| 2 | `pipeline_hash()` | `SemanticAwarePythonHasher` → JSON + SHA-256 | `ContentHash` | Unique per pipeline topology + schema; data-exclusive |
| 3 | Schema hash (system tag column naming) | `SemanticAwarePythonHasher.hash_object((tag_schema, data_schema)).to_hex(n)` | Truncated hex `str` | Unique per `(tag_schema, data_schema)` pair; embedded in system tag column names |
| 4 | Default `source_id` | `StarfixArrowHasher.hash_table(table).to_hex(n)` | Truncated hex `str` | Unique per raw table content; used as source identifier when none is provided |
| 4 | Default `source_id` of an in-memory source | `"<source kind>:" + StarfixArrowHasher.hash_table(table).to_string()` | `str`, `<source kind>:<algorithm label>:<hex>` | Unique per source kind and raw table content; used as the source's name when none is given |
| 5 | Per-row `record_id` (system tag value) | `uuid.uuid5(NAMESPACE, f"{source_id}::{provenance_token}")` | `bytes` (16, UUID v5) | Deterministic per `(source_id, row_identity)`; stable across re-runs of the same source |
| 6 | Join system tag suffix | `stream.pipeline_hash().to_hex(n)` + `:{idx}` appended to column name | Column name suffix | Unique per `(input topology, canonical join position)`; encodes full join lineage in column name |
| 6b | Aggregating operator system tag fold | `uuid.uuid5(NAMESPACE, joined_member_hex)` for `record_id`; SHA-256 of the canonical JSON list of member values otherwise | `bytes` (16) / hex `str`, plus a column name suffix | Deterministic per ordered member set; keeps system tags scalar through a many-to-one reduction |
Expand Down Expand Up @@ -65,8 +65,8 @@ Where a hash appears, and in which form:
| Form | Where |
|---|---|
| Labelled value | `__input_data_hash`, `__output_data_hash`, `__node_content_hash`, `__pipeline_base_entry_id`, the pipeline table's record id, the side-effect log's `record_id` and `record_id_hash`, the record id (`_record_hash`) of operator-node and cached-source tables, the `_content_hash` column, the hashes in a saved job file, `InvocationContext.invocation_hash` |
| Bare digest in a name | the function `uri` (its second component), the path components `schema:<pipeline digest>`, `instance:<content digest>` and `node:<content digest>`, system-tag column names, the operator `uri`, the default source id built from a table hash, the function `uri` at the start of a function output's source-info token |
| Labelled hash inside a source id | the default source id of a `DerivedSource` (`derived:<label>:<hex>`) and of `node.as_source()` (`<node path>:<label>:<hex>`) |
| Bare digest in a name | the function `uri` (its second component), the path components `schema:<pipeline digest>`, `instance:<content digest>` and `node:<content digest>`, system-tag column names, the operator `uri`, the function `uri` at the start of a function output's source-info token |
| Labelled hash inside a source id | every default source id built from a hash, `<source kind>:<label>:<hex>`: an in-memory source (`arrow_table:`, `dict:`, `list:`, `data_frame:`, then the labelled table hash), and a `DerivedSource`, which is also what `node.as_source()` returns (`derived:`, then the node's labelled content hash). `hash_source_id` in `core/sources/source_ids.py` is the single definition of that form. A `DerivedSource` built with `tags_as_data` has its own kind, `tags_as_data:<label>:<hex>` of its derivation hash (`DerivedSource._default_source_id`) |
| Digest only, inside a hash input | a nested hash in a structure: `{"__type__": "hash", "digest": "<hex>"}`; an extension-typed cell in a table: `b"<extension name>;<hex of the extension metadata>::" + digest` (the logical type's whole identifier, which the hashed field also keeps as its extension name and metadata, so the type is part of the hash of a column with no value too); the input data hash in the entry-id preimage (§3); the node content hash in the operator-node record hash; the leaf digests of the run snapshot hash (site 13) |

A function node's table in v0.3, with the values it stores:
Expand Down Expand Up @@ -320,7 +320,7 @@ The table below shows what each class returns from `identity_structure()`.
| `DataFunctionBase` | `self.uri` — `(canonical_function_name, output_schema_hash, major_version, data_function_type_id)` | `output_schema_hash` is site 11; see §5 |
| `FunctionPod` | `self.data_function` (bare object) when no ctx arg; `(self.data_function, self._ctx_arg_name)` when ctx arg present | `ctx_arg_name` is included when present so a ctx-aware pod has a distinct `content_hash` from a plain pod using the same data function |
| `ArrowTableStream` | `(producer, argument_symmetry(upstreams))` | Falls back to `(class_name, as_table(all_info=True), tag_columns)` when no producer or producer is not a `PipelineElementProtocol` |
| `RootSource` (`ArrowTableSource`) | `(class_name, (tag_schema, data_schema), source_id)` | `source_id` is the explicit user-supplied value; when not provided it defaults to the Arrow hash of the raw table (site 4). Data-inclusive base case of the Merkle chain. |
| `RootSource` (`ArrowTableSource`) | `(class_name, (tag_schema, data_schema), source_id)` | `source_id` is the explicit user-supplied value; when not provided it defaults to `<source kind>:<algorithm label>:<hex>` of the Arrow hash of the raw table (site 4). Data-inclusive base case of the Merkle chain. |
| `DerivedSource` | Origin node's `content_hash` | Inherits its generating node's identity |
| `DerivedSource` with `tags_as_data` | `(class_name, output_schema, source_id)`; the default `source_id` is `tags_as_data:<label>:<hex>` of the hash of `(("DerivedSource", sorted tags_as_data items), (origin,))`, the origin entering through its `content_hash`; a `record_id_column`, when set, is appended to the parameters as `("record_id_column", name)` | The rule of every root source; the origin and the mapping enter through the default `source_id`, which is the content hash an operator node doing the derivation would have |
| Operators (unary) | `(operator_class_name, upstream_stream)` | Stream reference resolved via `content_resolver` |
Expand Down Expand Up @@ -486,7 +486,7 @@ When a source is built by `SourceStreamBuilder`, four hash-related operations ha
sequence before the stream is returned:

1. **Site 3 — Schema hash:** computed from `(tag_schema, data_schema)`; embedded in system tag column names.
2. **Site 4 — Default `source_id`:** derived from the raw table hash if no explicit `source_id` is given.
2. **Site 4 — Default `source_id`:** `<source kind>:<algorithm label>:<hex>` of the raw table hash if no explicit `source_id` is given.
3. **Site 5 — Per-row `record_id`:** a UUID v5 computed from `(source_id, row_provenance_token)` and stored per row.
4. Two system tag columns, `_tag_source_id::{schema_hash}` and `_tag_record_id::{schema_hash}`, are appended to every row.

Expand All @@ -512,9 +512,9 @@ topology suffix, encoding the full join lineage into each column name.
| Field | Value |
|---|---|
| **Inputs** | Full raw Arrow table (all columns, before any system tag injection) |
| **Algorithm** | `StarfixArrowHasher.hash_table(table).to_hex(path_n_char)` — starfix hash format 2: the 3-byte format number `00 00 02` followed by a SHA-256 digest (`starfix` 0.5, byte-identical to the Rust crate) |
| **Output format** | Truncated hex `str` (length = `path_n_char`) |
| **Uniqueness guarantee** | Unique per raw table content; changes if any cell value changes |
| **Algorithm** | `hash_source_id(<source kind>, StarfixArrowHasher.hash_table(table), hexdigits=path_n_char)` — the table hash is starfix hash format 2: the 3-byte format number `00 00 02` followed by a SHA-256 digest (`starfix` 0.5, byte-identical to the Rust crate) |
| **Output format** | `str`: `<source kind>:<algorithm label>:<hex>`, for example `dict:arrow_v1:000002fe…`. The source kind is the class's config type key (`SOURCE_KIND`: `arrow_table`, `dict`, `list`, `data_frame`); the label is the one the hash carries. The kind and the label are never shortened; `hashing.path_n_char`, when set, shortens the hex only. The parts hold no `:`, so the id never contains the `::` that separates the parts of a source-info value |
| **Uniqueness guarantee** | Unique per source kind and raw table content; changes if any cell value changes or the rows are reordered |
| **Known exclusions** | Used **only** as a fallback when no explicit `source_id` is provided. A user-supplied `source_id` bypasses this computation entirely. |

---
Expand Down
14 changes: 7 additions & 7 deletions src/orcapod/core/nodes/function_node.py
Original file line number Diff line number Diff line change
Expand Up @@ -2255,6 +2255,10 @@ def get_all_records(
def as_source(self, tags_as_data: "Mapping[str, str] | None" = None):
"""Return a DerivedSource backed by the DB records of this node.

Its ``source_id`` is the ``DerivedSource`` default: without
``tags_as_data``, ``derived:<algorithm label>:<hex>`` of this node's
content hash.

Args:
tags_as_data: Maps a tag column of this node to the name of a data
column that receives a copy of it. When given, the result is a
Expand All @@ -2272,15 +2276,11 @@ def as_source(self, tags_as_data: "Mapping[str, str] | None" = None):

from orcapod.core.sources.derived_source import DerivedSource

source_id: str | None = None
if tags_as_data is None:
path_str = "/".join(self.node_identity_path)
# A source id is the source's name: a default built from a hash keeps
# the hash's label (developer, November 11, 2026).
source_id = f"{path_str}:{self.content_hash().to_string()}"
# No source_id is given: the source takes the DerivedSource default
# (``derived:<algorithm label>:<hex>`` of this node's content hash, or the
# ``tags_as_data`` id when a mapping is given).
return DerivedSource(
origin=self,
source_id=source_id,
tags_as_data=tags_as_data,
data_context=self.data_context_key,
config=self.orcapod_config,
Expand Down
Loading
Loading