Repository navigation
perf(pipeline): batched Delta commits make PipelineJob runs linear in rows (ITL-919 part 2) #306
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
b2a3a69
79141a5
c7ffc7d
e29890d
9a1ae63
5a06764
3c75c09
2e238c6
b727e0c
ce32c9e
3785d71
7d96a01
e9c78b5
a87cfd5
ff2c546
486db1d
12ef298
becdf14
7123a20
d32c84d
4b415a9
8f9adf8
286323c
07891f4
1fe07c5
27c39ea
c6d36ef
b8bcc81
5658c5a
561f599
b4387c0
65c2bd9
ff90d33
61b2278
06071a0
7552149
01cad57
e2b3db3
934417e
f6a3c3a
0369cfc
b55a3e6
488e2a2
8f282b8
4f0b4f1
65c6330
cf029bb
ffe4448
51dc905
11efe63
753b985
56b8b29
a94d067
cb7b39f
f9b2c6c
7ba60eb
a2ac8d9
ba396d4
c35bdf8
6b7d441
c2a5371
06ecbd5
0b29ea4
5aa4794
54bda57
ab8723c
00e724f
1f73fd1
aafb6f1
48678ca
5702cd8
f2c45c1
14c1546
f84217b
f9fb422
b226e09
186898d
335780d
144112e
4de96f4
04d3d71
1af8505
3eaabda
e76d98a
3fa5701
6a331c9
f21a189
fb70e9b
ddadddd
eb45dff
44819cb
97aa5a0
80c8368
2902490
591ca09
d26bdb8
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| ### Fixed | ||
|
|
||
| #### Delta reads return every row the database's handles wrote (ITL-864) | ||
|
|
||
| Right after a first `PipelineJob` run on a `DeltaTableDatabase`, `node.get_all_records()` could | ||
| return only some of the rows (1 of 2 or 3) on both orchestrators. Handles made with `at()` share | ||
| their pending writes, and any handle's flush writes all of them, but each handle kept its own | ||
| loaded copy of the table, so a sibling kept reading the version it had loaded before the flush. | ||
| The loaded tables and the stored-id sets are now shared by every handle of one root. Every read | ||
| method (`get_all_records`, `get_record_by_id`, `get_records_by_ids`, | ||
| `get_records_with_column_value`) returns what a flush would leave: the stored rows plus the | ||
| pending rows whose record id is not stored, or `None` when there are none. Before, | ||
| `get_records_by_ids` ignored pending rows, `get_records_with_column_value` ignored stored rows | ||
| while a pending batch existed, and a pending row whose id was already stored could be returned | ||
| twice or instead of the stored row. Writes made by another process or another root on the same | ||
| path are still seen only after the table is reloaded. (ITL-864 #293) |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| ### Fixed | ||
|
|
||
| #### A script that read a local Delta store no longer hangs at exit (ITL-892) | ||
|
|
||
| A process that had read a Delta table through `DeltaTableDatabase` or `DeltaTableSource` could | ||
| finish its work and then never exit: it hung in Arrow's thread-pool destructor, or, on Linux with | ||
| Python 3.11 or 3.12, aborted with `terminate called without an active exception`. It happened in | ||
| up to three runs in four on macOS and about one in ten on Linux, on every supported pyarrow and | ||
| deltalake version. The cause is upstream (apache/arrow#50482, apache/arrow#54536). deltalake's | ||
| default reader goes through a filesystem written in Python, and Arrow can release one of its | ||
| buffers on a worker thread while the interpreter is shutting down. | ||
|
|
||
| Orcapod now reads local Delta tables through a native pyarrow filesystem, which returns the same | ||
| schema and values. Tables on cloud storage (`s3://`, `gs://`, `az://`) still use deltalake's | ||
| reader and stay exposed until pyarrow ships the fix; see | ||
| [Troubleshooting](docs/troubleshooting.md). |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| ### Added | ||
|
|
||
| - **`PipelineJob.run(commit_every_rows=100)`** — how many rows a function node processes | ||
| between commits of its results and pipeline records (an `int >= 1`). Nodes also commit when | ||
|
Comment on lines
+3
to
+4
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Document the store batch-size limit.
|
||
| they finish, before a failed run raises, and at the end of the job; `1` commits after every | ||
| row. A hard crash on a durable store loses at most the rows the running node processed | ||
| since its last commit, and the rerun computes only those. (ITL-919) | ||
|
|
||
| ### Fixed | ||
|
|
||
| #### A `PipelineJob` on an in-memory store runs in time linear in its rows (ITL-919) | ||
|
|
||
| A `PipelineJob` on an `InMemoryArrowDatabase` runs in time linear in its rows. Every row a | ||
| function node processed used to filter, copy or rebuild the whole stored table (the | ||
| result lookup, the recomputation-index lookup, the duplicate check and the flush, each | ||
| over a table that gained one chunk per row), so 1,000 rows took about a minute. Lookups | ||
| now use hash indexes over the stored rows and a flush appends without copying them; | ||
| stored rows, ids and hashes are unchanged. | ||
|
|
||
| #### A `PipelineJob` on a Delta store runs in time linear in its rows (ITL-919) | ||
|
|
||
| A `PipelineJob` on a `DeltaTableDatabase` committed every row of every function node as | ||
| its own Delta version (a MERGE that reads the whole table) and re-read the stored table for | ||
| each per-row lookup, so 1,000 rows took about eight minutes. Function nodes inside | ||
| `job.run()` now commit every `commit_every_rows` rows (default 100, see *Added*); a batch | ||
| whose record ids are all new is appended instead of merged; and lookups are answered from | ||
| an indexed in-memory copy of the committed rows, read once and extended with each commit. | ||
| Stored rows, record ids, hashes, paths and schemas are unchanged. Behaviour changes: a store | ||
| failure that only a commit reveals (an unreadable table, a failed commit) surfaces at the | ||
| node's next commit point and names the node but no row, and the rows computed since the | ||
| last commit are computed again on the next run; `commit_every_rows=1` restores per-row | ||
| surfacing. A `DeltaTableDatabase` object keeps answering from the table version it loaded | ||
| until its own next commit (as before; it no longer re-reads the table per lookup). | ||
|
|
||
| #### An asynchronous run whose node fails raises instead of hanging (ITL-932) | ||
|
|
||
| On the asynchronous orchestrator, a node failure while more rows were in flight than the | ||
| channel buffers hold (about 16 to 64 rows) hung the run forever: the failure cancelled the | ||
| readers, and closing a full channel waited for room that never came. Closing a channel no | ||
| longer blocks; readers still receive every item sent before the close. | ||
|
|
||
| #### An asynchronous rerun of an ephemeral-result node recomputes its rows (ITL-923) | ||
|
|
||
| Rerunning a node with `NodeConfig(is_result_ephemeral=True)` on the asynchronous | ||
| orchestrator, after its ephemeral results were gone, neither recomputed nor recorded the | ||
| rows, and reading the node afterwards raised `EmptyDataAccessError`. It now recomputes them, | ||
| as the synchronous orchestrator does. | ||
|
|
||
| #### A failed side-effect completion lookup names the row | ||
|
|
||
| A `StoreReadError` from a side-effect pod's check of earlier deliveries now names the pod | ||
| and the row, as a failed write of the delivery record does. | ||
Uh oh!
There was an error while loading. Please reload this page.