Orcapod is a Python library for building reproducible scientific data pipelines. Data enters a pipeline through sources and flows through typed, immutable streams, in which every row is a tag (the keys that identify the row) and its data. Function pods compute new values from each row's data with plain Python functions; operators join, filter, group and reshape streams without computing new values. A job runs these steps against a database: every function result is stored, and a later run computes only what is not stored yet. Every row records which source and which source row it came from, and every value carries a provenance token.
Orcapod supports Python 3.11, 3.12 and 3.13 and is published on PyPI as orcapod:
The recommended way to install it is uv: uv add adds
Orcapod to a uv project (start one with uv init in a new directory), and uv run runs
your code in that project's environment:
uv add orcapodIn an environment not managed by uv, use pip install orcapod instead. A plain
uv add orcapod installs the latest stable release; to try a release candidate, name it,
for example uv add "orcapod==0.4.0rc1".
0.3.0 is the first release on PyPI since 0.1.4; 0.2.0 was never published to PyPI. Tables written by 0.2.0 or earlier are not read by 0.3.0: see Upgrading from 0.2.0 (it also covers 0.1.4).
Optional features are installed as extras, for example uv add "orcapod[spiraldb]":
| Extra | Adds |
|---|---|
orcapod[ray] |
Distributed execution with RayExecutor |
orcapod[redis] |
RedisCacher, a Redis-backed hash cache |
orcapod[postgresql] |
The PostgreSQL connector, table source and file hash cache |
orcapod[spiraldb] |
The SpiralDB connector and table source |
orcapod[torch] |
PyTorch, for torch.Tensor values (any installed torch works) |
orcapod[all] |
All of the above |
A source of two rows, one function pod, and a job that stores its results in a database:
from orcapod import PipelineJob, function_pod
from orcapod.databases import InMemoryArrowDatabase
from orcapod.sources import DictSource
@function_pod(output_keys="temp_fahrenheit")
def to_fahrenheit(temperature: float) -> float:
return temperature * 9.0 / 5.0 + 32.0
rows = [
{"experiment": "exp_001", "temperature": 20.5},
{"experiment": "exp_002", "temperature": 22.3},
]
database = InMemoryArrowDatabase()
def run_job(rows):
source = DictSource(data=rows, tag_columns=["experiment"], source_id="lab_results")
job = PipelineJob(name="lab", database=database)
with job: # record the steps; nothing runs yet
to_fahrenheit.pod(source, label="fahrenheit")
print(job.run())
return job
job = run_job(rows)
# Run of job 'lab': all rows succeeded.
# fahrenheit: 2 rows: 2 computed, 0 from store, 0 failed
print(job.nodes["fahrenheit"].as_table().sort_by("experiment").to_pylist())
# [{'experiment': 'exp_001', 'temp_fahrenheit': 68.9}, {'experiment': 'exp_002', 'temp_fahrenheit': 72.14}]Add a row and run a new job over the same database. The two stored results are found again by their input values, and only the new row is computed:
rows.append({"experiment": "exp_003", "temperature": 19.8})
job = run_job(rows)
# Run of job 'lab': all rows succeeded.
# fahrenheit: 3 rows: 1 computed, 2 from store, 0 failedInMemoryArrowDatabase keeps results for the life of the Python process;
DeltaTableDatabase("some/directory") writes them to disk, so they are found again in a new
session.
- Getting Started: sources, streams and function pods, step by step.
- Tutorial: a self-paced course in eleven short parts, from a first stream to a stored, re-runnable pipeline over files, with exercises.
- Concepts: sources, streams, function pods, operators, pipelines and jobs, identity and hashing, file types, arrays and your own data types.
- Reference: Hashing and Golden hashes.
- Troubleshooting: common errors, their causes and fixes.
- CHANGELOG: what changed in each release, with upgrade notes.
Orcapod is before version 1.0. In 0.3.0:
- Hash values are frozen for data context v1. Hashes name stored tables and find stored results again, so a hash value released in 0.3.0 does not change in a later release of the same data context; CI checks this. See Golden hashes.
- Library value types are provisional. The hashes of
numpy.ndarray,torch.Tensor, and pandas and polarsDataFrameandSeriesvalues are provisional until the library value type audit (after 0.4.0rc1) and may change then; results computed from them may then be computed again. Using one issuesProvisionalHashWarningonce per type per process. - Stores from earlier versions are not read as current. Tables written by 0.1.x or 0.2.0 are neither read as results nor modified, and results stored by a development build of 0.3 are not found again: 0.3.0 computes them again. Read Upgrading from 0.2.0 before running a job over an old store, in particular one with side-effect nodes.
- Connector databases cannot hold function nodes yet. A job over a SQLite, PostgreSQL or
SpiralDB database refuses every function node when it is built. Use
InMemoryArrowDatabaseorDeltaTableDatabasefor jobs.
SpikeInterface support lives in a separate package,
orcapod-extension-spikeinterface.
It is not published on PyPI; install it from its repository.
The project uses uv:
uv sync --all-extras --devEvery pull request must pass these checks:
uv run --with ruff ruff check .
uv run --with ruff ruff format --check .
uv run --with basedpyright basedpyright
uv run pytest tests/CI runs the tests in parallel (pytest-xdist). Locally they run serially; uv run pytest -n 4 tests/
runs them on four workers, each holding its own copy of the suite's memory, so use it only on a
machine with cores and memory to spare.
The PostgreSQL integration tests are skipped unless asked for
(uv run pytest -m postgres --postgres) and need a running PostgreSQL server, reached
through the standard PG* environment variables; see
the workflow that runs them.
Releases are cut by the GitHub Actions Release workflow; see
RELEASING.md.
Orcapod is released under the MIT License.