Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 20 additions & 1 deletion .zed/rules
Original file line number Diff line number Diff line change
Expand Up @@ -301,6 +301,7 @@ src/orcapod/
column_selection.py — Select/Drop Tag/Data columns
mappers.py — MapTags, MapData (rename columns)
filters.py — PolarsFilter
drop_nulls.py — DropNulls (drop rows with a null in named or all columns, T | None → T)
hashing/
semantic_hashing/ — BaseSemanticHasher, type handlers
semantic_types/ — Type conversion (Python ↔ Arrow)
Expand Down Expand Up @@ -367,7 +368,8 @@ FunctionNode.

### Strict operator / function pod boundary

Operators: inspect tags (never data content), can rename columns, cannot synthesize values.
Operators: inspect tags (never data content, except filters; see below), can rename columns,
cannot synthesize values.
Function Pods: inspect data content (never tags), synthesize new values, cached by content.

No data element comes from a tag value (developer, November 11, 2026, ITL-955). An operator
Expand All @@ -380,6 +382,11 @@ GroupBy is the only operator that reduces row count many→one: it collapses N r
tag tuple into one row with list-valued members. It synthesizes no new data values and keys
only on tags, never on data. Its non-key tags stay tags, list-valued, as Batch keeps them.

Filters are the operators that read data columns (ITL-943): PolarsFilter evaluates its
predicate on values, and DropNulls reads only whether a value is present (its validity),
never the value. A filter decides which rows exist and never synthesizes a value; no other
operator reads data content.

Operator or derived source? (developer, November 11, 2026) The test is: does any data
element come from a tag value, or from anywhere other than the input's data elements? If yes,
the step is not an operator: it is a derived source. So a tag's value is offered as data only
Expand Down Expand Up @@ -532,6 +539,18 @@ data. The rules:
(circular — it delegates back to the pod) and NOT against as_table() (legitimately differs:
output_schema ignores columns.source, so _source_*, _content_hash and _context_key appear in
the table but never in the schema).
- DropNulls keeps the rows where every named column (tag or data) is non-null and clears
the nullable flag of each named field, so T | None becomes T. Only top-level validity is
read (not list items, struct children or NaN). Its identity is the sorted tuple of unique
names. Both paths apply one rule, DropNulls._narrow_table: an Arrow validity mask and
the same arrays re-wrapped under the new field flags, with no Polars round-trip and no cast
other than large normalization. A named column that is already non-nullable is a no-op.
- DropNulls() with no argument (columns=None) means every user tag and data column of
the input (developer, November 12, 2026). The names are resolved from each input
(DropNulls._target_columns: stream.keys(), or the row's tag and data keys on the
async path) and never stored, so its identity is the fixed ("DropNulls", None),
different from any list of names. System-tag, source-info, context and meta columns are
never included. An explicitly empty collection is not "no argument": it raises ValueError.
- GroupBy requires every column in `by` to be a scalar tag column; raises InputValidationError
for unknown, data, or list-valued columns (list-valued usually means the stream came from
Batch — group before batching, not after). Members sort by non-key tag values with system
Expand Down
22 changes: 20 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -312,7 +312,8 @@ src/orcapod/
│ ├── group_by.py # GroupBy (many→one reduction keyed on tag values)
│ ├── column_selection.py # Select/Drop Tag/Data columns
│ ├── mappers.py # MapTags, MapData (rename columns)
│ └── filters.py # PolarsFilter
│ ├── filters.py # PolarsFilter
│ └── drop_nulls.py # DropNulls (drop rows with a null in named or all columns, T | None → T)
├── hashing/
│ └── semantic_hashing/ # BaseSemanticHasher, type handlers
├── semantic_types/ # Type conversion (Python ↔ Arrow)
Expand Down Expand Up @@ -384,7 +385,7 @@ FunctionNode.

| | Operator | Function Pod |
|---|---|---|
| Inspects data content | Never | Yes |
| Inspects data content | Never, except filters (see below) | Yes |
| Inspects / uses tags | Yes | No |
| Can rename columns | Yes | No |
| Synthesizes new values | No | Yes |
Expand All @@ -401,6 +402,11 @@ tag tuple into one row with list-valued members. It still synthesizes no new dat
every emitted element came from an input row — and it keys only on tags, never on data. Its
non-key tags stay tags, list-valued, as `Batch` keeps them.

Filters are the operators that read data columns (ITL-943): `PolarsFilter` evaluates its
predicate on values, and `DropNulls` reads only whether a value is present (its validity),
never the value. A filter decides which rows exist and never synthesizes a value; no other
operator reads data content.

**Operator or derived source?** (developer, November 11, 2026) The test is: does any data
element come from a tag value, or from anywhere other than the input's data elements? If yes,
the step is not an operator: it is a derived source. So a tag's value is offered as data only
Expand Down Expand Up @@ -576,6 +582,18 @@ data. The rules:
`as_table()` (which legitimately differs: `ArrowTableStream.output_schema` ignores
`columns.source`, so `_source_*`, `_content_hash`, and `_context_key` appear in the table
but never in the schema).
- `DropNulls` keeps the rows where every named column (tag or data) is non-null and clears
the nullable flag of each named field, so `T | None` becomes `T`. Only top-level validity is
read (not list items, struct children or NaN). Its identity is the sorted tuple of unique
names. Both paths apply one rule, `DropNulls._narrow_table`: an Arrow validity mask and
the same arrays re-wrapped under the new field flags, with no Polars round-trip and no cast
other than large normalization. A named column that is already non-nullable is a no-op.
- `DropNulls()` with no argument (`columns=None`) means every user tag and data column of
the input (developer, November 12, 2026). The names are resolved from each input
(`DropNulls._target_columns`: `stream.keys()`, or the row's tag and data keys on the
async path) and never stored, so its identity is the fixed `("DropNulls", None)`,
different from any list of names. System-tag, source-info, context and meta columns are
never included. An explicitly empty collection is not "no argument": it raises `ValueError`.
- `GroupBy` requires every column in `by` to be a scalar tag column; it raises
`InputValidationError` for unknown, data, or list-valued columns (the last usually means the
stream came from `Batch` — group before batching, not after). Members are sorted by non-key
Expand Down
21 changes: 21 additions & 0 deletions changelog.d/ITL-943.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
### Added

- **`DropNulls` operator and `stream.drop_nulls(columns)`** — keeps the rows where
every named column is non-null and narrows each named column from `T | None` to `T`, in
`output_schema()` and in the Arrow table, so a pod whose parameter is a plain `T` can
consume a column that is nullable by specification. The names may be tag columns, data
columns or both; a name that is not a column of the stream raises `InputValidationError`,
and a column that is already non-nullable is left as it is. With no argument
(`DropNulls()`, `stream.drop_nulls()`) the columns are every tag and data column of the
input stream: a row with a null in any of them is dropped and all of them become
non-nullable. System-tag, source-info, context and meta columns are never included. On a
source whose columns are all nullable (SpiralDB stores every column nullable) this form can
drop most rows; naming the columns is the precise form. An empty collection is not "no
argument" and raises `ValueError`. Only the top-level value is
tested (a null list or struct; not list items, struct fields or NaN). Other columns, system
tags, source info and row order are unchanged. The operator's identity is the set of names;
the no-argument form has one fixed identity, different from that of any list of names.
A surviving row's data hash (or tag hash, for a tag column) changes when the named column
was nullable, because nullability is part of the row's Arrow schema; it becomes the hash
the row has in a stream that declared the column non-nullable. No existing hash changes.
(ITL-943)
35 changes: 35 additions & 0 deletions design_issues/ITL-943-DN1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
## `src/orcapod/core/operators/drop_nulls.py` — null rows and nullable columns (ITL-943)

### DN1 — No way to turn a `T | None` column into `T`
**Status:** resolved
**Severity:** medium
**Issue:** ITL-943

Many value columns are nullable by specification (EDI, Spiral tables), and a pod written as
`def f(x: int)` is refused on an `int | None` column. `PolarsFilter` could drop the null rows,
but the column stayed `T | None`, so the pod was still refused; a nullable key column likewise
could not be joined with a non-nullable one.

**Fix:** `DropNulls(columns)` and `stream.drop_nulls(columns)` keep the rows where every
named column (tag or data) is non-null and clear the nullable flag of each named field.
Decisions (coordinator, November 11, 2026): one operator for tag and data columns; only
top-level validity is read; a missing column is `InputValidationError`; an already non-nullable
column is a no-op; identity is the sorted tuple of unique names; sync and async paths share one
table rule (`_narrow_table`), pure Arrow. A surviving row's `Data` (or `Tag`) content hash
changes when a named column was nullable, to the hash of the same row declared non-nullable;
no existing golden value moved.

No-argument form (developer, November 12, 2026: "plain no argument = drop all is fair and as
applied to both tags and data columns"): `DropNulls()` / `stream.drop_nulls()` tests and
narrows every user tag and data column of the input. The names are resolved from each input
(`stream.keys()`; per row on the async path) and never stored, so the identity is the fixed
`("DropNulls", None)` and one operator object works on any stream. `DropNulls()` and
`DropNulls([every column])` give the same rows and schema but are different identities.
System-tag, source-info, context and meta columns are never included (meta columns are
filtered by prefix, because a stream built straight from a table lists them among its data
keys, ITL-846). An explicitly empty collection still raises `ValueError`: an empty list is a
different intent from no argument, and a computed list that came out empty must not drop rows.

Open: the class and method names (`DropNulls` / `drop_nulls` versus `DropNulls` /
`drop_nulls`) await the developer's decision. Possible follow-up: a variant that raises on a
null instead of dropping the row.
149 changes: 146 additions & 3 deletions docs/concepts/operators.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ This separation is a core Orcapod design principle:

| | Operator | Function Pod |
|---|---|---|
| Inspects data content | Never | Yes |
| Inspects data content | Never, except filters (see below) | Yes |
| Inspects / uses tags | Yes | No |
| Can rename columns | Yes | No |
| Synthesizes new values | No | Yes |
Expand All @@ -32,6 +32,10 @@ One rule holds for every operator: **no data element comes from a tag value**. A
use tags to join, filter, group or order rows, but it never turns a tag column into a data
column, so a function pod never receives tag content.

The filters are the operators that read data columns: `PolarsFilter` evaluates its predicate
on values, and `DropNulls` reads only whether a value is present (its validity), never the
value. Both decide which rows exist; neither synthesizes a value.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

## Operator categories

### `UnaryOperator` -- single input
Expand Down Expand Up @@ -370,6 +374,145 @@ except ValueError as exc:
# Cannot hash the Polars expression 'col("age").python_udf()' for an Orcapod identity
```

### DropNulls

`DropNulls(columns)` keeps the rows where every named column is non-null, and narrows
each named column from `T | None` to `T`. `DropNulls()`, with no names, does this for every
tag and data column (see [Every column](#every-column) below). Use it when a column is nullable by specification
but a function needs a value: a pod whose parameter is a plain `int` refuses an `int | None`
column, and `PolarsFilter` can drop the null rows but leaves the type as it was.

```python
import pyarrow as pa
from orcapod import function_pod
from orcapod.operators import DropNulls
from orcapod.sources import ArrowTableSource

recordings = ArrowTableSource(
pa.table(
{
"session": pa.array(["s1", "s2", "s3"], pa.large_string()),
"start_ms": pa.array([1000, None, 3000], pa.int64()),
},
schema=pa.schema(
[
pa.field("session", pa.large_string(), nullable=False),
pa.field("start_ms", pa.int64(), nullable=True),
]
),
),
tag_columns=["session"],
)
print(recordings.output_schema()[1])
# Schema({'start_ms': int | None})


@function_pod(output_keys="start_s")
def to_seconds(start_ms: int) -> float:
return start_ms / 1000


try:
to_seconds.pod(recordings)
except ValueError as exc:
print(str(exc).rsplit(": ", 1)[1])
# column 'start_ms' has type int | None, the function expects int

present = DropNulls("start_ms")(recordings)
print(present.output_schema()[1])
# Schema({'start_ms': int})
print(to_seconds.pod(present).as_table().to_pylist())
# [{'session': 's1', 'start_s': 1.0}, {'session': 's3', 'start_s': 3.0}]
```

The rules:

- **Columns.** Pass one name or a collection of names; a row survives only if every named
column is non-null. The names may be tag columns, data columns, or both. A name that is
not a tag or data column of the stream raises `InputValidationError`. With no argument
the columns are every tag and data column of the input. An empty collection is not "no
argument": it raises `ValueError`, so a list of names that came out empty never drops
rows by surprise.
- **What counts as null.** Only the top-level value: a null list or a null struct is null. An
empty list, a list that holds nulls, a struct whose fields are null and a floating-point
NaN are not. Nested fields keep their nullability.
- **Already non-nullable columns** are accepted and left as they are, so applying the
operator twice is the same as applying it once.
- **Everything else is unchanged**: other columns' types and nullability, logical types
(a `Path | None` column becomes `Path`), system tags, source info and row order. A result
with no rows has the same narrowed schema.
- **Identity** is the set of names: `DropNulls("a")`, `DropNulls(["a"])` and
`DropNulls({"a"})` are one operator, and the order of names does not matter.
`DropNulls()` has one fixed identity of its own, whatever stream it is applied to.
- **Row hashes.** Nullability is part of a row's Arrow schema, so a surviving row's data
hash changes when a named data column was nullable (its tag hash, when a named tag column
was). The new hash is the one the row has in a stream that declared the column
non-nullable from the start.

Tag columns work the same way. A stream whose key column is `str | None` cannot be joined
with one whose key is `str` (`Join` never reconciles types), so narrow the key first. The
stream method is `drop_nulls`:

```python
by_method = recordings.drop_nulls(["session", "start_ms"])
by_operator = DropNulls(["start_ms", "session"])(recordings)
print(by_method.pipeline_hash() == by_operator.pipeline_hash())
# True
```

#### Every column

`DropNulls()` and `stream.drop_nulls()`, with no names, drop every row that has a null in
**any** tag or data column, and make every tag and data column non-nullable. System-tag,
source-info, context and meta columns are never tested or changed.

```python
visits = ArrowTableSource(
pa.table(
{
"session": pa.array(["s1", "s2", "s3", "s4"], pa.large_string()),
"start_ms": pa.array([1000, None, 3000, 4000], pa.int64()),
"notes": pa.array([None, "noisy", None, "ok"], pa.large_string()),
}
),
tag_columns=["session"],
)
print(visits.output_schema())
# (Schema({'session': str | None}), Schema({'start_ms': int | None, 'notes': str | None}))

complete = visits.drop_nulls()
print(complete.output_schema())
# (Schema({'session': str}), Schema({'start_ms': int, 'notes': str}))
print(complete.as_table().to_pylist())
# [{'session': 's4', 'start_ms': 4000, 'notes': 'ok'}]

timed = visits.drop_nulls("start_ms")
print(timed.as_table().to_pylist())
# [{'session': 's1', 'start_ms': 1000, 'notes': None}, {'session': 's3', 'start_ms': 3000, 'notes': None}, {'session': 's4', 'start_ms': 4000, 'notes': 'ok'}]
```

!!! warning "The no-argument form can drop most of a table"

A null in any tag or data column drops the row, including a column that is empty for a
good reason. Above, `notes` is empty for most sessions, so three of the four rows are
gone although only one of them lacks `start_ms`. Sources whose columns are all nullable
are common: SpiralDB stores every column as nullable. For such a source, name the columns
a downstream step needs (`drop_nulls("start_ms")`); that is the precise form. Use the
no-argument form only when every column must be present.

The column names are read from each input stream when the operator is applied, so one
`DropNulls()` object works on any stream. On the same stream it gives the rows and schema
of `DropNulls([every tag and data column])`, but the two are different operators: their
identities, and so the pipeline and content hashes of their output streams, differ.

```python
by_name = DropNulls(["session", "start_ms", "notes"])(visits)
print(complete.as_table() == by_name.as_table())
# True
print(complete.pipeline_hash() == by_name.pipeline_hash())
# False
```

### Pick and Index

`Pick(column, key)` extracts one entry from a struct- or dict-typed data column, and
Expand Down Expand Up @@ -464,8 +607,8 @@ operators or function pods.

Every stream also exposes the operators as methods, which reads well when chaining:
`join`, `semi_join`, `batch`, `group_by`, `select_tag_columns`, `select_data_columns`,
`drop_tag_columns`, `drop_data_columns`, `map_tags`, `map_data`, `polars_filter`, `pick` and
`index`.
`drop_tag_columns`, `drop_data_columns`, `map_tags`, `map_data`, `polars_filter`,
`drop_nulls`, `pick` and `index`.

```python
import polars as pl
Expand Down
2 changes: 2 additions & 0 deletions src/orcapod/core/operators/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
SelectDataColumns,
SelectTagColumns,
)
from .drop_nulls import DropNulls
from .filters import PolarsFilter
from .group_by import GroupBy
from .index import Index
Expand All @@ -27,6 +28,7 @@
"DropTagColumns",
"DropDataColumns",
"PolarsFilter",
"DropNulls",
"Pick",
"Index",
]
Loading
Loading