diff --git a/superpowers/plans/2026-11-13-itl-1001-crash-recovery-flake-after-347.md b/superpowers/plans/2026-11-13-itl-1001-crash-recovery-flake-after-347.md new file mode 100644 index 000000000..0f2122038 --- /dev/null +++ b/superpowers/plans/2026-11-13-itl-1001-crash-recovery-flake-after-347.md @@ -0,0 +1,110 @@ +# ITL-1001 crash-recovery flake after PR #347 Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use sensei:subagent-driven-development (recommended) or sensei:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** `test_crash_mid_batch_keeps_committed_batches[async]` proves that committed batches survive a crash mid-batch, with no assumption about the order in which rows finish and no wait that can expire under load. + +**Architecture:** Test-only change in `tests/test_pipeline/test_job_crash_recovery.py`. The assertions compare sets of rows read from the store (rows with a committed pipeline record, rows with a committed result) with the rows the rerun computes. A new async case forces a row before 200 to finish after the second commit, using the same event as the crash row and no clock. + +**Tech Stack:** pytest, subprocess job script, deltalake. + +--- + +## Cause (established 2026-11-13) + +CI job 131810636077 (`pytest (3.13)`, run 43403086940, PR #346 head `126ef033`) failed at +`test_job_crash_recovery.py:228`: `min(calls) >= 200` was false. Every earlier assertion +held: crash exit 3 (so the bounded wait of #347 did not time out), `records == 200`, +`200 <= results < 234`, rerun exit 0, `len(calls) == 250 - results`. + +Category (b): the event was set and the counts were right. The failure is the test's +assumption that the two committed batches are input rows 0..199. + +- `FunctionPod.async_execute` (`src/orcapod/core/function_pod.py`) starts one task per + row. Each one runs the sync function on the default thread pool + (`data_function.py`, `run_in_executor(None, ...)`), and sends its result on completion. +- `FunctionNode`'s record stage (`record_and_forward` → `_row_processed`) takes rows in + completion order and commits every `commit_every_rows` rows. +- So the committed batches are the first 200 rows to finish. A row before 200 that + finishes after 200 others (thread preemption on a loaded runner) is recorded after the + second commit. Its result is still pending at the crash, and the rerun computes it. + +The library guarantee holds. `CachedFunctionPod.async_process_data` stores the result +(pending) before the row reaches the record stage, and `_commit_pending_writes` commits +results before pipeline records. So every committed record has a committed result. +Nothing at or after the crash row is committed, and the rerun computes exactly the rows +without a committed result. + +Deterministic repro (scratch script: row 199 waits on the same "second commit done" event +as the crash row): records 200, results 207, rerun `min(calls) == 199`. The current +assertion fails 3/3; the order-free check holds 3/3. + +Latent off-by-one: `results < 234` should be `<= 234`. Rows 0..233 are 234 rows, and all +of them may have committed results. + +## Decisions + +- The in-child wait becomes unbounded (`Event.wait()`), and exit code 4 goes. It cannot + deadlock. The executor queue is FIFO, so rows at or after the crash row start only after + every earlier row has started. The default pool has at least 5 threads, so the one + blocked late row leaves room. `run_job`'s `subprocess.run(timeout=600)` stays as the + hang guard. It is not a timing assumption of the test. +- The late-row case runs on async only. On the sync path, rows run in order, so a waiting + row 199 would block the second commit forever. +- No library change, no changelog fragment (test only, as #347), no design_issues entry. + +## Out of scope (report in worker_done) + +These are the six interrupt tests in `tests/test_pipeline/test_job_run_in_event_loop.py` +(`test_interrupt_while_running_inside_a_loop_stops_the_run`, +`test_second_interrupt_returns_and_a_late_function_writes_nothing[*]` and +`test_second_interrupt_names_an_async_function_blocking_the_loop[*]`). Their time bounds: +- `threading.Timer(1.0, _thread.interrupt_main)`: the second press assumes the first has + been handled and the run is waiting for the stuck function within 1 s. +- `time.sleep(0.05)` per row with `calls_when_stopped < N_SLOW_ROWS`, then a + `time.sleep(0.2)` quiet window. +- `stuck_release.wait(timeout=30)` and the 30 s deadline loops are hang guards only. + +Making them event-driven needs an observable "run is waiting for the stuck function" state +from the library. That is not a few lines, so it is not in this PR. + +### Task 1: Failing regression case (late row) + +**Files:** +- Modify: `tests/test_pipeline/test_job_crash_recovery.py` + +- [ ] Add a seventh script argument `late` ("none" or a row number). In `double`, when + `crash_kind == "fn"` and `x == late`, wait on `earlier_batches_committed`. Add + `late: int | None = None` to `run_job`, passed as `str(late)` or `"none"`. +- [ ] Parametrize the crash test over `sync`, `async`, `async-late-row` (async, late 199) + with the old assertions. Run it: `async-late-row` fails at `min(calls) >= 200`. +- [ ] WIP commit. + +### Task 2: Order-free assertions, unbounded wait + +- [ ] Add `stored_rows(store, kind) -> set[int]`: `k` of the pipeline table, `y // 2` of + the result table. +- [ ] The crash test asserts: + - two whole batches: 200 pipeline records on 200 distinct rows, each with a committed + result; + - nothing at or after the crash row stored (`max(computed) < 234`); + - sync: `recorded == computed == set(range(200))`; + - late row: the late row is neither recorded nor computed; + - the rerun computes exactly the rows without a committed result: + `sorted(calls) == sorted(set(range(N_ROWS)) - computed)`; + - `stable(store) == uninterrupted`. +- [ ] Make the wait `earlier_batches_committed.wait()`, and update the script comment and + the module docstring. +- [ ] Run the file on 3.12 and 3.13. Loop: before (origin/main test, late row injected) + vs after, ≤ 50 iterations, sequential. +- [ ] ruff format/check, basedpyright, commit, full suite, push, PR (`Refs ITL-1001`), Copilot review. + +## Progress (2026-11-13) + +- Task 1 done: `async-late-row` failed with the old assertions (`assert False` at the + `min(calls) >= 200` line, as on CI); the other three cases passed. +- Task 2 done: 4/4 pass on 3.12 and 3.13; ruff clean; basedpyright exit 0, no new + warnings. +- Before (origin/main test, row 199 late, scratch repro): old assertion failed 20/20; + order-free check held 20/20 (records 200, results 207, rerun min 199). +- After: 25/25 sequential runs of the two async cases passed. diff --git a/tests/test_pipeline/test_job_crash_recovery.py b/tests/test_pipeline/test_job_crash_recovery.py index 188a9af17..4388ac003 100644 --- a/tests/test_pipeline/test_job_crash_recovery.py +++ b/tests/test_pipeline/test_job_crash_recovery.py @@ -5,7 +5,9 @@ subprocess, as a power loss or ``kill -9`` would) on a ``DeltaTableDatabase``: - in the middle of a batch: exactly the committed batches are stored, and the rerun - computes only the rows that were not; + computes only the rows that were not. On the async path rows finish, and so are + recorded and committed, in any order: a batch holds the rows that finished first, not + the first rows of the input; - between a batch's result commit and its pipeline-record commit, also in a batch that starts with rows whose results were already stored: the results are kept, the rerun computes nothing for them and writes their pipeline records. @@ -19,7 +21,7 @@ import sys import textwrap from pathlib import Path -from typing import Any +from typing import Any, cast import deltalake import pyarrow as pa @@ -30,11 +32,11 @@ N_ROWS = 250 # Runs one job over N_ROWS rows. Arguments: store path, calls file, commit_every_rows, -# orchestrator ("sync" or "async"), crash ("none", "fn:" or "flush:"). -# "fn:" exits while computing row x, once the batches of the rows before x are -# committed; "flush:" exits on the n-th store-wide flush. The run's first flush -# commits what was pending before it; each later one is a node's commit, called right -# after the node committed its result table. +# orchestrator ("sync" or "async"), crash ("none", "fn:" or "flush:"), late row +# ("none" or a row number). "fn:" exits while computing row x, once the batches of the +# rows before x are committed; "flush:" exits on the n-th store-wide flush. The run's +# first flush commits what was pending before it; each later one is a node's commit, +# called right after the node committed its result table. # # The async orchestrator computes rows on a thread pool ahead of the stage that records # them and commits every commit_every_rows rows, so without a wait row x could be @@ -42,9 +44,17 @@ # after it wait until the store has finished the run's first flush and one flush per # batch before x, so the crash falls in the batch that holds x, after the earlier # batches are committed and before the result of any row from x on is stored. On the -# sync path the rows come in order and the wait returns at once. The wait is bounded: if -# the commits never come, the job exits with code 4, which run_job rejects, so the test -# fails instead of hanging. +# sync path the rows come in order and the wait returns at once. +# +# The pool's rows also finish in any order, and the node records them in the order they +# finish. A late row (with "fn:" only, async only) waits for the same commits, so it +# finishes after the batches before x are committed without it: a row of the input's +# first batches that a loaded machine finishes late, made deterministic. +# +# The wait has no time bound, because it cannot deadlock: the pool starts rows in input +# order, so rows from x on only take threads once every row before x has started, and +# all of those return except the one late row. run_job's subprocess timeout is the hang +# guard if a change ever breaks that. JOB_SCRIPT = textwrap.dedent( """ import os @@ -57,7 +67,7 @@ from orcapod.pipeline.async_orchestrator import AsyncPipelineOrchestrator from orcapod.sources import DictSource - store_path, calls_path, every, orchestrator, crash = sys.argv[1:6] + store_path, calls_path, every, orchestrator, crash, late = sys.argv[1:7] crash_kind, _, crash_arg = crash.partition(":") earlier_batches_committed = threading.Event() if crash_kind == "fn": @@ -65,13 +75,7 @@ def wait_for_earlier_batches() -> None: - if not earlier_batches_committed.wait(timeout=120): - print( - "the batches before the crash row were never committed", - file=sys.stderr, - flush=True, - ) - os._exit(4) + earlier_batches_committed.wait() @function_pod(output_keys="y") @@ -82,6 +86,8 @@ def double(x: int) -> int: wait_for_earlier_batches() if x == int(crash_arg): os._exit(3) + if crash_kind == "fn" and late != "none" and x == int(late): + wait_for_earlier_batches() return 2 * x @@ -125,7 +131,12 @@ def __getattr__(self, name: str) -> Any: def run_job( - tmp_path: Path, store: Path, every: int, orchestrator: str, crash: str = "none" + tmp_path: Path, + store: Path, + every: int, + orchestrator: str, + crash: str = "none", + late: int | None = None, ) -> tuple[int, list[int]]: """Run the job in a subprocess; return its exit code and the rows it computed.""" script = tmp_path / "job.py" @@ -134,7 +145,7 @@ def run_job( calls.write_text("") done = subprocess.run( [sys.executable, str(script), str(store), str(calls), str(every)] - + [orchestrator, crash], + + [orchestrator, crash, "none" if late is None else str(late)], capture_output=True, text=True, timeout=600, @@ -165,6 +176,21 @@ def row_count(store: Path, kind: str) -> int: return 0 if table is None else table.num_rows +def stored_rows(store: Path, kind: str) -> list[int]: + """The input rows (``x``) of the committed rows of the ``kind`` table, sorted. + + A pipeline record names its row by the tag ``k``, which equals ``x``; a result + holds ``y = 2 * x``. + """ + table = read(tables(store).get(kind)) + if table is None: + return [] + # Both columns are non-null int64; to_pylist is typed as possibly None. + if kind == "pipeline": + return sorted(cast("list[int]", table["k"].to_pylist())) + return sorted(y // 2 for y in cast("list[int]", table["y"].to_pylist())) + + def stable(store: Path) -> dict[str, list[tuple[object, ...]]]: """The stored columns that an interrupted and an uninterrupted run must share. @@ -205,27 +231,45 @@ def uninterrupted(tmp_path_factory: pytest.TempPathFactory) -> dict[str, list[An return stable(tmp / "store") -@pytest.mark.parametrize("orchestrator", ["sync", "async"]) +@pytest.mark.parametrize( + ("orchestrator", "late"), + [("sync", None), ("async", None), ("async", 199)], + ids=["sync", "async", "async-late-row"], +) def test_crash_mid_batch_keeps_committed_batches( - tmp_path: Path, orchestrator: str, uninterrupted: dict[str, list[Any]] + tmp_path: Path, + orchestrator: str, + late: int | None, + uninterrupted: dict[str, list[Any]], ) -> None: - """Killed at row 234 with 100 rows per commit: 200 rows stored, 50 recomputed.""" + """Killed at row 234 with 100 rows per commit: two whole batches stored. + + The rerun computes exactly the rows without a committed result. On the async path + the two batches are the 200 rows that finished first, whichever they are; the + late-row case makes row 199 finish after the second commit. + """ store = tmp_path / "store" - code, _ = run_job(tmp_path, store, 100, orchestrator, crash="fn:234") + code, _ = run_job(tmp_path, store, 100, orchestrator, crash="fn:234", late=late) assert code == 3 - results, records = row_count(store, "result"), row_count(store, "pipeline") + recorded = stored_rows(store, "pipeline") + computed = stored_rows(store, "result") + # Two whole batches: 200 rows with a pipeline record, each with a committed result + # (results are committed before their pipeline records). + assert len(set(recorded)) == len(recorded) == 200 + assert set(recorded) <= set(computed) + assert len(set(computed)) == len(computed) + # Nothing from the crash row on is stored. + assert max(computed) < 234 if orchestrator == "sync": - assert (results, records) == (200, 200) - else: - # Results are committed before their pipeline records, so results of rows - # still on their way to being recorded may be committed too. - assert records == 200 and 200 <= results < 234 + assert recorded == computed == list(range(200)) + if late is not None: + # The late row finished after the second commit, so neither is stored. + assert late not in computed code, calls = run_job(tmp_path, store, 100, orchestrator) assert code == 0 - # Only rows without a committed result are computed again. - assert len(calls) == N_ROWS - results - assert min(calls) == results if orchestrator == "sync" else min(calls) >= 200 + # Exactly the rows without a committed result are computed again. + assert sorted(calls) == sorted(set(range(N_ROWS)) - set(computed)) assert stable(store) == uninterrupted