Repository navigation
feat(operators): DropNulls drops rows with a null in named columns and narrows them to T (ITL-943) #311
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
eywalker
merged 14 commits into
main
from
eywalker/itl-943-operator-that-drops-rows-where-given
Nov 12, 2026
Merged
feat(operators): DropNulls drops rows with a null in named columns and narrows them to T (ITL-943) #311
Changes from all commits
Commits
Show all changes
14 commits
Select commit
Hold shift + click to select a range
4f99b12
docs(plans): ITL-943 DropNullRows operator plan
36cf709
feat(operators): add DropNullRows, which drops rows with a null in na…
630b7cf
docs(operators): document DropNullRows; add golden entries, changelog…
2a2f898
test(operators): type-clean DropNullRows tests (ITL-943)
2bef2da
fix(operators): DropNullRows keeps null-typed columns and handles str…
4c1df1b
refactor(operators): inline DropNullRows validity mask (ITL-943)
4d43c9b
docs(operators): address CodeRabbit review of DropNullRows (ITL-943)
7479bc5
refactor(operators)!: rename DropNullRows to DropNulls and drop_null_…
be11eb6
docs(changelog): move the ITL-943 entry to a changelog fragment (ITL-…
138c53a
Merge remote-tracking branch 'origin/main' into eywalker/itl-943-oper…
fad6832
docs(design-issues): move the ITL-943 entries to design_issues/ files…
1ac1e0e
Merge remote-tracking branch 'origin/main' into eywalker/itl-943-oper…
8cffc5a
feat(drop_nulls): no argument means every tag and data column (ITL-943)
f999ef6
Merge remote-tracking branch 'origin/main' into eywalker/itl-943-oper…
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| ### Added | ||
|
|
||
| - **`DropNulls` operator and `stream.drop_nulls(columns)`** — keeps the rows where | ||
| every named column is non-null and narrows each named column from `T | None` to `T`, in | ||
| `output_schema()` and in the Arrow table, so a pod whose parameter is a plain `T` can | ||
| consume a column that is nullable by specification. The names may be tag columns, data | ||
| columns or both; a name that is not a column of the stream raises `InputValidationError`, | ||
| and a column that is already non-nullable is left as it is. With no argument | ||
| (`DropNulls()`, `stream.drop_nulls()`) the columns are every tag and data column of the | ||
| input stream: a row with a null in any of them is dropped and all of them become | ||
| non-nullable. System-tag, source-info, context and meta columns are never included. On a | ||
| source whose columns are all nullable (SpiralDB stores every column nullable) this form can | ||
| drop most rows; naming the columns is the precise form. An empty collection is not "no | ||
| argument" and raises `ValueError`. Only the top-level value is | ||
| tested (a null list or struct; not list items, struct fields or NaN). Other columns, system | ||
| tags, source info and row order are unchanged. The operator's identity is the set of names; | ||
| the no-argument form has one fixed identity, different from that of any list of names. | ||
| A surviving row's data hash (or tag hash, for a tag column) changes when the named column | ||
| was nullable, because nullability is part of the row's Arrow schema; it becomes the hash | ||
| the row has in a stream that declared the column non-nullable. No existing hash changes. | ||
| (ITL-943) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| ## `src/orcapod/core/operators/drop_nulls.py` — null rows and nullable columns (ITL-943) | ||
|
|
||
| ### DN1 — No way to turn a `T | None` column into `T` | ||
| **Status:** resolved | ||
| **Severity:** medium | ||
| **Issue:** ITL-943 | ||
|
|
||
| Many value columns are nullable by specification (EDI, Spiral tables), and a pod written as | ||
| `def f(x: int)` is refused on an `int | None` column. `PolarsFilter` could drop the null rows, | ||
| but the column stayed `T | None`, so the pod was still refused; a nullable key column likewise | ||
| could not be joined with a non-nullable one. | ||
|
|
||
| **Fix:** `DropNulls(columns)` and `stream.drop_nulls(columns)` keep the rows where every | ||
| named column (tag or data) is non-null and clear the nullable flag of each named field. | ||
| Decisions (coordinator, November 11, 2026): one operator for tag and data columns; only | ||
| top-level validity is read; a missing column is `InputValidationError`; an already non-nullable | ||
| column is a no-op; identity is the sorted tuple of unique names; sync and async paths share one | ||
| table rule (`_narrow_table`), pure Arrow. A surviving row's `Data` (or `Tag`) content hash | ||
| changes when a named column was nullable, to the hash of the same row declared non-nullable; | ||
| no existing golden value moved. | ||
|
|
||
| No-argument form (developer, November 12, 2026: "plain no argument = drop all is fair and as | ||
| applied to both tags and data columns"): `DropNulls()` / `stream.drop_nulls()` tests and | ||
| narrows every user tag and data column of the input. The names are resolved from each input | ||
| (`stream.keys()`; per row on the async path) and never stored, so the identity is the fixed | ||
| `("DropNulls", None)` and one operator object works on any stream. `DropNulls()` and | ||
| `DropNulls([every column])` give the same rows and schema but are different identities. | ||
| System-tag, source-info, context and meta columns are never included (meta columns are | ||
| filtered by prefix, because a stream built straight from a table lists them among its data | ||
| keys, ITL-846). An explicitly empty collection still raises `ValueError`: an empty list is a | ||
| different intent from no argument, and a computed list that came out empty must not drop rows. | ||
|
|
||
| Open: the class and method names (`DropNulls` / `drop_nulls` versus `DropNulls` / | ||
| `drop_nulls`) await the developer's decision. Possible follow-up: a variant that raises on a | ||
| null instead of dropping the row. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.