Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
114 commits
Select commit Hold shift + click to select a range
fc22de6
test(referential_tags): failing EDI-shaped end-to-end test (ITL-1023)
Nov 13, 2026
02b6235
feat(referential_tags): WIP representation, sources, select/drop, Joi…
Nov 13, 2026
c3c5e99
feat(referential_tags): row-preserving ops, renames, guards, async Jo…
Nov 13, 2026
c42eece
feat(referential_tags): renames per developer decision; EDI test on t…
Nov 13, 2026
880dcd4
Merge remote-tracking branch 'origin/main' into eywalker/itl-1023-wor…
Nov 13, 2026
853da82
test(referential_tags): existing tests pin plain tags where they asse…
Nov 13, 2026
1ec232e
feat(pipeline): saved job and pipeline format 0.3.0; v0.3 files load …
Nov 13, 2026
6120671
Merge remote-tracking branch 'origin/main' into eywalker/itl-1023-wor…
Nov 13, 2026
860045e
test(referential_tags): pin plain tags in main's torch golden case an…
Nov 13, 2026
6a6d72b
feat(function_pod): referential tags fall through a pod call (ITL-1023)
Nov 13, 2026
ed51400
fix(referential_tags): fall-through keeps exact types and returned ob…
Nov 13, 2026
5832772
Merge remote-tracking branch 'origin/main' into eywalker/itl-1023-wor…
Nov 13, 2026
3e12d62
fix(referential_tags): type-check clean; fall-through pod refuses to …
Nov 13, 2026
d8628af
test(pipeline): commit the saved-job fixture's CSV and Parquet files …
Nov 13, 2026
1fd2daa
test(objective): pin plain tags where the objective tests assume v0.3…
Nov 13, 2026
34fb2a0
Merge remote-tracking branch 'origin/main' into eywalker/itl-1023-wor…
Nov 13, 2026
4473e2e
Merge remote-tracking branch 'origin/main' into eywalker/itl-1023-wor…
Nov 13, 2026
4c0c687
docs(plan): ITL-1023 status log
Nov 13, 2026
5607af1
docs(plan): ITL-1017 merged source-info token plan
Nov 13, 2026
5e8028e
fix(referential_tags): one merged source-info token for a value merge…
Nov 13, 2026
aaff7da
docs(referential_tags): merged source-info token, changelog, open poi…
Nov 13, 2026
8d7e3c9
Merge remote-tracking branch 'origin/main' into eywalker/itl-1023-wor…
Nov 13, 2026
c78ba80
docs(plan): ITL-1017 status log
Nov 13, 2026
8c7cfb6
Merge origin/main into the referential tags slice (ITL-1023, ITL-1017)
Nov 14, 2026
0bfd72a
Merge origin/main (ITL-1074) into the referential tags slice (ITL-1023)
Nov 14, 2026
634ad40
fix(group_by): a referential non-key tag keeps its data column as a l…
Nov 14, 2026
99b5737
Merge origin/main (4dd015a4) into the referential tags slice (ITL-1023)
Nov 16, 2026
404066e
fix(referential_tags): main's new paths meet referential tags (ITL-1023)
Nov 16, 2026
177d5ad
fix(referential_tags): GroupBy key order, meta stripping and join err…
Nov 16, 2026
61129ae
fix(referential_tags): zero-row heads keep references; pin main's new…
Nov 16, 2026
f359aa0
docs: main's new pages pin plain tags on the referential tags slice (…
Nov 16, 2026
3e0ae5a
feat(join): a referential key that differs only in nullability joins …
Nov 16, 2026
22e7662
test(referential_tags): type-clean key nullability assertion (ITL-1023)
Nov 16, 2026
157632b
test: pin plain tags in main's remaining v0.3-shape suites (ITL-1023)
Nov 16, 2026
84d1312
feat(referential_tags): stored tables and saved jobs keep tag binding…
Nov 16, 2026
199221c
feat(referential_tags): refuse a database that cannot keep the marker…
Nov 16, 2026
8f7d50a
ci(coderabbit): review pull requests that target any base branch, on …
eywalker Nov 16, 2026
7d6c580
chore: merge the slice branch (.coderabbit.yaml, PR #405) into ITL-1019
Nov 16, 2026
fc1db0a
fix(pipeline): a saved-node error of a loaded job names the job file …
Nov 16, 2026
00818bd
fix(pipeline): refuse a saved source config whose bindings contradict…
Nov 16, 2026
ec5984a
fix(pipeline): check an explicitly empty tag_columns against the slot…
Nov 16, 2026
26d981a
feat(referential_tags): operator nodes store and derived sources pres…
Nov 16, 2026
cb3dbc4
refactor(operator_node): the stored-table check returns nothing; one …
Nov 16, 2026
3c51256
test(referential_tags): reach the DerivedSource binding check; log IT…
Nov 16, 2026
cecbaf3
feat(databases): a database states keeps_field_metadata; an undeclare…
Nov 16, 2026
25ff7db
feat(batch): Batch on referential tags (ITL-1017) — into the referent…
eywalker Nov 16, 2026
67435e0
feat(referential_tags): SelectColumns/DropColumns on referential tags…
eywalker Nov 16, 2026
57aaa5c
Merge origin/main (37716c76, ITL-1154) into the referential tags slic…
Nov 16, 2026
c111ca4
feat(referential_tags): stored tables and saved jobs keep tag binding…
eywalker Nov 16, 2026
69776b0
chore: merge the slice branch (PR #402 squash-merged) into ITL-1019 W3
Nov 16, 2026
40c57da
Merge the slice head c111ca47 (#402, ITL-1019) into the ITL-1154 refr…
Nov 16, 2026
7137e72
fix(databases): a SpiralDB retry publishes metadata whose write faile…
Nov 17, 2026
1e9565a
fix(databases): a SpiralDB write that stores no row never replaces st…
Nov 17, 2026
5261e9a
test(referential_tags): type-clean field metadata in the SpiralDB sto…
Nov 17, 2026
ded5305
fix(databases): SpiralDB writes a table's Arrow metadata before its r…
Nov 17, 2026
f030aab
fix(databases): a failed first SpiralDB write removes the metadata it…
Nov 17, 2026
edc98fd
Merge origin/main (62899bda: #408, #409, #410, #411) into the referen…
Nov 17, 2026
4b49640
fix(databases): a failed SpiralDB write puts the metadata entry back …
Nov 17, 2026
a6cf552
fix(databases): read the SpiralDB metadata snapshot before the nullab…
Nov 17, 2026
ac68895
Merge origin/main (7c735b7a, v0.3.0 release commit) into the referent…
Nov 17, 2026
01f3db4
docs(changelog): concurrent SpiralDB writers are not coordinated; fol…
Nov 17, 2026
6274a3c
chore: merge the refreshed slice (main 7c735b7a, v0.3.0) into ITL-101…
Nov 17, 2026
a7f08ed
test(referential_tags): meet main's SpiralDB column refusal and hash-…
Nov 17, 2026
ce32310
fix(operator_node): replayed rows carry the node's data context in th…
Nov 17, 2026
8b73ba7
feat(operators): TagBy / stream.tag_by makes data columns referential…
Nov 17, 2026
6aa2314
docs(referential-tags): describe the three moves by role after tag_by…
Nov 17, 2026
e037e58
Merge pull request #407 from metamorphic-labs/eywalker/itl-1019-w3-de…
eywalker Nov 17, 2026
e91d256
Merge remote-tracking branch 'origin/eywalker/itl-1023-workshop-slice…
Nov 17, 2026
43bb0ef
Merge origin/main (23fd8f2a: #414, #415, #416) into the referential t…
Nov 17, 2026
eb32ab1
test(tag_by): a logged TagBy operator node stores its new tag marked …
Nov 17, 2026
9c48452
chore(changelog): fold the slice's fragments into changelog.d/ITL-102…
Nov 17, 2026
a6b4a17
docs(referential-tags): select and drop act on a referential tag by r…
Nov 17, 2026
3ab814e
refactor(referential_tags): remove ReferenceMaterializationWarning (I…
Nov 17, 2026
194254e
Merge remote-tracking branch 'origin/eywalker/itl-1023-workshop-slice…
Nov 17, 2026
8cd398f
fix(tag_by): keep the input's column layout and data context (ITL-1175)
Nov 17, 2026
424f731
Merge branch 'eywalker/itl-1175-operator-and-stream-method-to-mark' i…
Nov 17, 2026
5292a60
revert(referential_tags): keep ReferenceMaterializationWarning (ITL-1…
Nov 17, 2026
1e585a7
Merge origin/main (565201b2: #417) into the referential tags slice (I…
Nov 17, 2026
27fee5a
feat(merge_join): handle referential tags as Join does (ITL-1194)
Nov 17, 2026
83ff606
fix(tag_by): keep each row's data context key; correct the spec on LO…
Nov 17, 2026
35579f8
Merge branch 'eywalker/itl-1175-operator-and-stream-method-to-mark' i…
Nov 17, 2026
01c3b72
docs(design_issues): log ST5, the two tag orders of ArrowTableStream …
Nov 17, 2026
447597b
Merge branch 'eywalker/itl-1175-operator-and-stream-method-to-mark' i…
Nov 17, 2026
50b580b
fix(referential_tags): address the CodeRabbit review of the slice (IT…
Nov 17, 2026
016052a
docs(referential_tags): log the deferred review findings (ITL-1199 to…
Nov 17, 2026
fc205da
docs(referential-tags): sort the MergeJoin sample's rows before print…
Nov 17, 2026
5dd0634
Merge remote-tracking branch 'origin/eywalker/itl-1023-workshop-slice…
Nov 17, 2026
378eca2
Merge branch 'eywalker/itl-1175-operator-and-stream-method-to-mark' i…
Nov 17, 2026
d7d1b34
Merge remote-tracking branch 'origin/eywalker/itl-1023-workshop-slice…
Nov 17, 2026
38478e0
Merge origin/main (232ee360: #418) into the referential tags slice (I…
Nov 17, 2026
05deee1
Merge remote-tracking branch 'origin/eywalker/itl-1023-workshop-slice…
Nov 17, 2026
a373ca6
Merge branch 'eywalker/itl-1175-operator-and-stream-method-to-mark' i…
Nov 17, 2026
09ffdf7
Merge remote-tracking branch 'origin/eywalker/itl-1023-workshop-slice…
Nov 17, 2026
0937242
Merge pull request #423 from metamorphic-labs/eywalker/itl-1194-merge…
eywalker Nov 17, 2026
460ff01
chore(changelog): fold the ITL-1194 fragment (#423) into changelog.d/…
Nov 17, 2026
816d8c8
Merge remote-tracking branch 'origin/eywalker/itl-1023-workshop-slice…
Nov 17, 2026
bd8f020
Merge branch 'eywalker/itl-1175-operator-and-stream-method-to-mark' i…
Nov 17, 2026
e43c609
docs(design_issues): log AB1, the async barrier input's column order …
Nov 17, 2026
c4cee17
Merge pull request #421 from metamorphic-labs/eywalker/itl-1195-opera…
eywalker Nov 17, 2026
b77cb01
refactor(data_tags): rename "referential tag" to "data-tag" (ITL-1023)
Nov 17, 2026
b011809
Merge origin/main (51be5e4e: #422 add_tags, #419) into the data-tags …
Nov 17, 2026
c2aafc4
fix(data_tags): data-tags come first among the data columns (ITL-1205)
Nov 17, 2026
20309a0
feat(add_tags): AddTags works on a stream with data-tags (ITL-1196, I…
Nov 17, 2026
0248fc0
fix(data_tags): async rows keep data-tags first; fall-through prevali…
Nov 17, 2026
8d6a1d9
docs(data_tags): answer the CodeRabbit doc findings on PR #342 (ITL-1…
Nov 17, 2026
3817334
Merge origin/main (7c8c5efb: #425 Starlight docs site) into the data-…
Nov 17, 2026
66d6ed7
fix(side_effects): pass a data-tag only to a parameter that takes a k…
Nov 17, 2026
99487ce
fix(data_tags): refuse an EmptyData hit of a fall-through node; selec…
Nov 17, 2026
ce427c4
Merge origin/main (ad3f2411: #424 pipeline diagram, #430 Graphviz rem…
Nov 17, 2026
b702ce4
fix(pick, index): only a data-tag is refused in place; a plain tag ge…
Nov 17, 2026
aee3791
docs: iterate a stream directly instead of calling iter_data() in loo…
Nov 17, 2026
bf13922
docs: make uv add orcapod the recommended install (ITL-1228)
Nov 17, 2026
5e0b1dd
Merge origin/main (9698dbcc: #342 data-tags slice, squashed) into ITL…
Nov 18, 2026
09ee49c
Merge origin/main (4a754eac: #428 CI job names) into 1224-DOCS
Nov 18, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 9 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,18 +13,24 @@ provenance token.

Orcapod supports Python 3.11, 3.12 and 3.13 and is published on PyPI as `orcapod`:

The recommended way to install it is [uv](https://docs.astral.sh/uv/): `uv add` adds
Orcapod to a uv project (start one with `uv init` in a new directory), and `uv run` runs
your code in that project's environment:

```bash
pip install orcapod
# or, in a uv project
uv add orcapod
```

In an environment not managed by uv, use `pip install orcapod` instead. A plain
`uv add orcapod` installs the latest stable release; to try a release candidate, name it,
for example `uv add "orcapod==0.4.0rc1"`.

0.3.0 is the first release on PyPI since 0.1.4; 0.2.0 was never published to PyPI. Tables
written by 0.2.0 or earlier are not read by 0.3.0: see
[Upgrading from 0.2.0](https://github.com/metamorphic-labs/orcapod-python/blob/main/CHANGELOG.md#upgrading-from-020)
(it also covers 0.1.4).

Optional features are installed as extras:
Optional features are installed as extras, for example `uv add "orcapod[spiraldb]"`:

| Extra | Adds |
|-------|------|
Expand Down
12 changes: 6 additions & 6 deletions docs/concepts/arrays.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ source = DictSource(
tag_columns=["trial"],
source_id="trials",
)
for tag, data in scale.pod(source).iter_data():
for tag, data in scale.pod(source):
print(tag["trial"], data["scaled"].dtype, data["scaled"].shape)
# t1 float32 (2, 3)
```
Expand Down Expand Up @@ -246,7 +246,7 @@ big = np.asfortranarray(np.arange(6, dtype=">f4").reshape(2, 3))
kept = DictSource(
data=[{"trial": "t1", "signal": big}], tag_columns=["trial"], source_id="kept"
)
for tag, data in kept.iter_data():
for tag, data in kept:
print(data["signal"].dtype.str, data["signal"].flags.f_contiguous)
# >f4 True
```
Expand All @@ -269,9 +269,9 @@ values have different hashes.
## PyTorch tensors

A `torch.Tensor` can be a data value too. PyTorch is an optional dependency. Tensor support
works whenever PyTorch can be imported, however it was installed: `pip install "orcapod[torch]"`
installs it from PyPI, or install the build that suits your hardware yourself (see
pytorch.org; on Linux the PyPI build includes CUDA, and pytorch.org also offers smaller
works whenever PyTorch can be imported, however it was installed: `uv add "orcapod[torch]"`
(or `pip install "orcapod[torch]"`) installs it from PyPI, or install the build that suits
your hardware yourself (see pytorch.org; on Linux the PyPI build includes CUDA, and pytorch.org also offers smaller
CPU-only builds). Orcapod needs PyTorch 2.6 or later, the first version whose restricted
loader (below) reads `uint16`, `uint32` and `uint64` tensors, and is tested with PyTorch 2.15. Orcapod
hashes a tensor's elements through NumPy (`Tensor.numpy()`), so your PyTorch build must
Expand Down Expand Up @@ -299,7 +299,7 @@ tensors = DictSource(
tag_columns=["trial"],
source_id="tensor_trials",
)
for tag, data in scale_tensor.pod(tensors).iter_data():
for tag, data in scale_tensor.pod(tensors):
print(tag["trial"], data["scaled"].dtype, tuple(data["scaled"].shape))
# t1 torch.float32 (2, 3)
```
Expand Down
2 changes: 1 addition & 1 deletion docs/concepts/data-tags.md
Original file line number Diff line number Diff line change
Expand Up @@ -227,7 +227,7 @@ column:
batched = spikes.batch()
print(batched.keys())
# (('unit', 'time'), ('unit', 'time', 'amp'))
for tag, data in batched.iter_data():
for tag, data in batched:
print(tag["time"], data["time"])
# [10, 20] [10, 20]
```
Expand Down
4 changes: 2 additions & 2 deletions docs/concepts/file-hash-caching.md
Original file line number Diff line number Diff line change
Expand Up @@ -265,8 +265,8 @@ enable_file_hash_caching(db_path=Path("shared/nfs/cache.db"))

### Using the Postgres backend

Requires `psycopg` (install with `pip install 'orcapod[postgresql]'`) and a reachable
Postgres server:
Requires `psycopg` (install with `uv add "orcapod[postgresql]"`, or
`pip install "orcapod[postgresql]"`) and a reachable Postgres server:

<!-- docs-test: skip -->
```python
Expand Down
4 changes: 2 additions & 2 deletions docs/concepts/file-types.md
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,7 @@ def summarize(recording: op.File) -> Path:
result = summarize.pod(files)
print(result.output_schema()[1])
# Schema({'summary': Path})
for tag, data in result.iter_data():
for tag, data in result:
print(tag.as_dict(), data.as_dict()["summary"].read_text().strip())
# {'name': 'a'} 3 lines
# {'name': 'b'} 2 lines
Expand Down Expand Up @@ -180,7 +180,7 @@ def total_lines(recording: list[op.File]) -> int:


everything = files.batch()
for _, data in total_lines.pod(everything).iter_data():
for _, data in total_lines.pod(everything):
print(data.as_dict())
# {'total_lines': 5}
```
Expand Down
16 changes: 8 additions & 8 deletions docs/concepts/function-pods.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ print("Data schema:", dict(data_schema))
# Data schema: {'bmi': <class 'float'>}

# Iterate over results
for tag, data in result.iter_data():
for tag, data in result:
print(f" {tag.as_dict()} -> {data.as_dict()}")
# {'subject_id': 'mouse_01'} -> {'bmi': 1756.9444444444446}
# {'subject_id': 'mouse_02'} -> {'bmi': 2209.9999999999995}
Expand Down Expand Up @@ -114,7 +114,7 @@ wide = DictSource(
)

result = compute_bmi.pod(wide) # 'age' is not a parameter: it is dropped
for tag, data in result.iter_data():
for tag, data in result:
print(f"{tag.as_dict()} -> {data.as_dict()}")
# {'subject_id': 'mouse_01'} -> {'bmi': 1756.9444444444446}
# {'subject_id': 'mouse_02'} -> {'bmi': 2209.9999999999995}
Expand Down Expand Up @@ -289,7 +289,7 @@ explicit:
def twice(weight: float) -> list[float]:
return list((weight, weight * 2))

for tag, data in twice.pod(weights).iter_data():
for tag, data in twice.pod(weights):
print(data.as_dict())
# {'weights': [25.3, 50.6]}
```
Expand Down Expand Up @@ -1131,12 +1131,12 @@ recordings = DictSource(
tag_columns=["session"], tag_only_columns=True,
)
readings = load_readings.pod(recordings)
for tag, data in readings.iter_data():
for tag, data in readings:
print(tag["session"], dict(data["readings"].schema))
# s1 {'sample': UInt16, 'voltage': Float32}
# s2 {'sample': UInt16, 'voltage': Float32}

for tag, data in count_rows.pod(readings).iter_data():
for tag, data in count_rows.pod(readings):
print(tag["session"], data["n_rows"])
# s1 2
# s2 3
Expand Down Expand Up @@ -1177,7 +1177,7 @@ labels = DictSource(
tag_columns=["sample"], tag_only_columns=True,
data_schema={"sample": str, "counts": list[int | None]},
)
for tag, data in labels.iter_data():
for tag, data in labels:
print(data["counts"])
# [3, None, 1]
```
Expand Down Expand Up @@ -1218,7 +1218,7 @@ heights = DictSource(
)

result = compute_bmi.pod(weights, heights)
for tag, data in result.iter_data():
for tag, data in result:
print(f"{tag.as_dict()} -> {data.as_dict()}")
# {'subject_id': 'mouse_01'} -> {'bmi': 1756.9444444444446}
# {'subject_id': 'mouse_02'} -> {'bmi': 2209.9999999999995}
Expand All @@ -1242,7 +1242,7 @@ chosen = DictSource(
)

result = compute_bmi.pod(weights, heights, chosen)
for tag, data in result.iter_data():
for tag, data in result:
print(f"{tag.as_dict()} -> {data.as_dict()}")
# {'subject_id': 'mouse_02'} -> {'bmi': 2209.9999999999995}
```
Expand Down
16 changes: 8 additions & 8 deletions docs/concepts/operators.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,7 @@ scale_b = DictSource(
)

merged = MergeJoin()(scale_a, scale_b)
for tag, data in merged.iter_data():
for tag, data in merged:
print(tag.as_dict(), "->", data.as_dict())
# {'subject_id': 'mouse_01'} -> {'weight': [24.9, 25.3]}
# {'subject_id': 'mouse_02'} -> {'weight': [22.1, 22.6]}
Expand Down Expand Up @@ -197,7 +197,7 @@ weighed = DictSource(
)

only_weighed = SemiJoin()(all_subjects, weighed)
for tag, data in only_weighed.iter_data():
for tag, data in only_weighed:
print(tag.as_dict(), "->", data.as_dict())
# {'subject_id': 'mouse_01'} -> {'age': 12}
# {'subject_id': 'mouse_02'} -> {'age': 8}
Expand All @@ -222,7 +222,7 @@ source = DictSource(

batch = Batch()
batched = batch.process(source)
for tag, data in batched.iter_data():
for tag, data in batched:
print("Tags:", tag.as_dict())
# Tags: {'subject_id': ['mouse_01', 'mouse_02']}
print("Data:", data.as_dict())
Expand Down Expand Up @@ -263,7 +263,7 @@ source = DictSource(
grouped = GroupBy(by=["subject_id"])(source)
print(grouped.keys())
# (('subject_id',), ('spike_count',))
for tag, data in grouped.iter_data():
for tag, data in grouped:
print("Tags:", tag.as_dict(), "Data:", data.as_dict())
# Tags: {'subject_id': 'mouse_01'} Data: {'spike_count': [1200, 980]}
# Tags: {'subject_id': 'mouse_02'} Data: {'spike_count': [1450]}
Expand Down Expand Up @@ -293,7 +293,7 @@ with_probe = DerivedSource(source, tags_as_data={"probe": "probe_name"})
by_subject = GroupBy(by=["subject_id"])(with_probe)
print(by_subject.keys())
# (('subject_id',), ('spike_count', 'probe_name'))
for tag, data in by_subject.iter_data():
for tag, data in by_subject:
print(tag.as_dict(), data.as_dict())
# {'subject_id': 'mouse_01'} {'spike_count': [1200, 980], 'probe_name': ['imec0', 'imec1']}
# {'subject_id': 'mouse_02'} {'spike_count': [1450], 'probe_name': ['imec0']}
Expand Down Expand Up @@ -455,7 +455,7 @@ from orcapod.operators import PolarsFilter

filt = PolarsFilter(predicates=[pl.col("age") > 10])
filtered = filt.process(source)
for tag, pkt in filtered.iter_data():
for tag, pkt in filtered:
print(f"{tag.as_dict()} -> {pkt.as_dict()}")
# {'subject_id': 'mouse_01'} -> {'age': 12, 'weight': 25.3}
# {'subject_id': 'mouse_03'} -> {'age': 15, 'weight': 27.8}
Expand Down Expand Up @@ -659,7 +659,7 @@ keyword arguments or as one dict (the only positional argument), as `map_tags` t
```python
labelled = sessions.add_tags(cohort="A", run=3)
print(labelled.keys()) # (('subject_id', 'session', 'cohort', 'run'), ('age', 'weight'))
for tag, data in labelled.iter_data():
for tag, data in labelled:
print(tag.as_dict(), "->", data.as_dict())
# {'subject_id': 'mouse_01', 'session': 1, 'cohort': 'A', 'run': 3} -> {'age': 12, 'weight': 25.3}
# {'subject_id': 'mouse_02', 'session': 2, 'cohort': 'A', 'run': 3} -> {'age': 8, 'weight': 22.1}
Expand Down Expand Up @@ -803,7 +803,7 @@ Every stream also exposes the operators as methods, which reads well when chaini
import polars as pl

adults = subjects.join(measurements).polars_filter(pl.col("age") > 10)
for tag, data in adults.iter_data():
for tag, data in adults:
print(tag.as_dict(), "->", data.as_dict())
# {'subject_id': 'mouse_01'} -> {'weight': 25.3, 'age': 12}
```
Expand Down
6 changes: 3 additions & 3 deletions docs/concepts/sources.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,8 +94,8 @@ pyspiral keeps the login in its per-user directory (`~/Library/Application Suppo
on macOS, `~/.config/pyspiral/` on Linux):

```bash
pip install "orcapod[spiraldb]"
spiral login
uv add "orcapod[spiraldb]" # or, without uv: pip install "orcapod[spiraldb]"
uv run spiral login
```

Log in before you build a source. Without a saved login, pyspiral starts a sign-in in the
Expand Down Expand Up @@ -634,7 +634,7 @@ print("Data keys:", data_keys)
# Data keys: ('age', 'weight')

# Iterate over (Tag, Data) pairs
for tag, data in source.iter_data():
for tag, data in source:
print(f" Tag: {tag.as_dict()}, Data: {data.as_dict()}")
# Tag: {'subject_id': 'mouse_01'}, Data: {'age': 12, 'weight': 25.3}
# Tag: {'subject_id': 'mouse_02'}, Data: {'age': 8, 'weight': 22.1}
Expand Down
10 changes: 6 additions & 4 deletions docs/concepts/streams.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,11 +78,13 @@ tag_keys, data_keys = stream.keys()

### `iter_data()`

Iterates over (Tag, Data) pairs. Each Tag and Data is an immutable datagram that you can
inspect with `.as_dict()`:
Iterates over (Tag, Data) pairs. A stream is itself iterable, and iterating it is the same as
calling `iter_data()`, so a loop is written `for tag, data in stream:`; the method is there
for when the iterator is passed on as a value (`list(stream.iter_data())`). Each Tag and Data
is an immutable datagram that you can inspect with `.as_dict()`:

```python
for tag, data in stream.iter_data():
for tag, data in stream:
print(tag.as_dict(), data.as_dict())
# {'subject_id': 'mouse_01'} {'age': 12, 'weight': 25.3}
# {'subject_id': 'mouse_02'} {'age': 8, 'weight': 22.1}
Expand Down Expand Up @@ -358,7 +360,7 @@ print("Data schema:", dict(data_schema))
# Data schema: {'age': <class 'int'>, 'weight': <class 'float'>}

# Iterate over (Tag, Data) pairs
for tag, data in source.iter_data():
for tag, data in source:
print(f" {tag.as_dict()} -> {data.as_dict()}")
# {'subject_id': 'mouse_01'} -> {'age': 12, 'weight': 25.3}
# {'subject_id': 'mouse_02'} -> {'age': 8, 'weight': 22.1}
Expand Down
20 changes: 17 additions & 3 deletions docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,19 @@ This guide walks you through the core workflow in Orcapod: creating a data sourc
inspecting its stream, applying a transformation with a function pod, and examining
the results.

## Installing

Install Orcapod with [uv](https://docs.astral.sh/uv/), the recommended way: start a project
with `uv init` (in a new directory), add Orcapod to it, and run your scripts with `uv run`:

```bash
uv init
uv add orcapod
uv run python my_pipeline.py
```

In an environment not managed by uv, `pip install orcapod` installs it instead.

## Creating a source

A **source** is the entry point for data in an Orcapod pipeline. The simplest way to
Expand Down Expand Up @@ -74,10 +87,11 @@ print(data_keys)

### Iterating over rows

Use `iter_data()` to walk through each (Tag, Data) pair:
A stream is iterable: a `for` loop walks through each (Tag, Data) pair. Iterating a stream
is the same as calling its `iter_data()` method:

```python
for tag, data in source.iter_data():
for tag, data in source:
print(f"Tag: {tag.as_dict()}, Data: {data.as_dict()}")
# Tag: {'experiment': 'exp_001'}, Data: {'temperature': 20.5, 'pressure': 1.01}
# Tag: {'experiment': 'exp_002'}, Data: {'temperature': 22.3, 'pressure': 0.98}
Expand Down Expand Up @@ -160,7 +174,7 @@ now reflects the function's output types.
Iterate over the results:

```python
for tag, data in result.iter_data():
for tag, data in result:
print(f"Tag: {tag.as_dict()}, Data: {data.as_dict()}")
# Tag: {'experiment': 'exp_001'}, Data: {'temp_fahrenheit': 68.9, 'is_high_pressure': True}
# Tag: {'experiment': 'exp_002'}, Data: {'temp_fahrenheit': 72.14, 'is_high_pressure': False}
Expand Down
12 changes: 9 additions & 3 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,18 @@ content-addressable, and reproducible by design.
Orcapod supports Python 3.11, 3.12 and 3.13 and is published on PyPI as `orcapod`, under the
MIT License:

The recommended way to install it is [uv](https://docs.astral.sh/uv/): `uv add` adds
Orcapod to a uv project (start one with `uv init` in a new directory), and `uv run` runs
your code in that project's environment:

```bash
pip install orcapod
# or, in a uv project
uv add orcapod
```

In an environment not managed by uv, use `pip install orcapod` instead. A plain
`uv add orcapod` installs the latest stable release; to try a release candidate, name it,
for example `uv add "orcapod==0.4.0rc1"`.

`0.3.0` is the first release on PyPI since `0.1.4`; `0.2.0` was never published to PyPI.
Tables written by `0.2.0` or earlier are not read by `0.3.0`: see
`CHANGELOG.md` in the repository, *Upgrading from 0.2.0*
Expand Down Expand Up @@ -47,7 +53,7 @@ def compute_birth_year(age: int) -> int:

# 3. Apply the function pod and inspect the output
result = compute_birth_year.pod(source)
for tag, data in result.iter_data():
for tag, data in result:
print(f"{tag.as_dict()} -> {data.as_dict()}")
# {'name': 'Alice'} -> {'birth_year': 1996}
# {'name': 'Bob'} -> {'birth_year': 2001}
Expand Down
3 changes: 3 additions & 0 deletions docs/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -314,3 +314,6 @@ Reading a Delta table yourself with `deltalake.DeltaTable(...).to_pyarrow_table(
are installing a version of Orcapod older than 0.2, which required the Graphviz
development headers. Install Orcapod 0.2 or newer.
- **Python version.** Orcapod needs Python 3.11 or newer.
- **A release candidate is not installed.** `uv add orcapod` (like `pip install orcapod`)
takes the latest stable release and skips pre-releases. Name the release candidate:
`uv add "orcapod==0.4.0rc1"`.
Loading
Loading