Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
# ITL-1001 crash-recovery flake after PR #347 Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use sensei:subagent-driven-development (recommended) or sensei:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** `test_crash_mid_batch_keeps_committed_batches[async]` proves that committed batches survive a crash mid-batch, with no assumption about the order in which rows finish and no wait that can expire under load.

**Architecture:** Test-only change in `tests/test_pipeline/test_job_crash_recovery.py`. The assertions compare sets of rows read from the store (rows with a committed pipeline record, rows with a committed result) with the rows the rerun computes. A new async case forces a row before 200 to finish after the second commit, using the same event as the crash row and no clock.

**Tech Stack:** pytest, subprocess job script, deltalake.

---

## Cause (established 2026-11-13)

CI job 131810636077 (`pytest (3.13)`, run 43403086940, PR #346 head `126ef033`) failed at
`test_job_crash_recovery.py:228`: `min(calls) >= 200` was false. Every earlier assertion
held: crash exit 3 (so the bounded wait of #347 did not time out), `records == 200`,
`200 <= results < 234`, rerun exit 0, `len(calls) == 250 - results`.

Category (b): the event was set and the counts were right. The failure is the test's
assumption that the two committed batches are input rows 0..199.

- `FunctionPod.async_execute` (`src/orcapod/core/function_pod.py`) starts one task per
row. Each one runs the sync function on the default thread pool
(`data_function.py`, `run_in_executor(None, ...)`), and sends its result on completion.
- `FunctionNode`'s record stage (`record_and_forward` → `_row_processed`) takes rows in
completion order and commits every `commit_every_rows` rows.
- So the committed batches are the first 200 rows to finish. A row before 200 that
finishes after 200 others (thread preemption on a loaded runner) is recorded after the
second commit. Its result is still pending at the crash, and the rerun computes it.

The library guarantee holds. `CachedFunctionPod.async_process_data` stores the result
(pending) before the row reaches the record stage, and `_commit_pending_writes` commits
results before pipeline records. So every committed record has a committed result.
Nothing at or after the crash row is committed, and the rerun computes exactly the rows
without a committed result.

Deterministic repro (scratch script: row 199 waits on the same "second commit done" event
as the crash row): records 200, results 207, rerun `min(calls) == 199`. The current
assertion fails 3/3; the order-free check holds 3/3.

Latent off-by-one: `results < 234` should be `<= 234`. Rows 0..233 are 234 rows, and all
of them may have committed results.

## Decisions

- The in-child wait becomes unbounded (`Event.wait()`), and exit code 4 goes. It cannot
deadlock. The executor queue is FIFO, so rows at or after the crash row start only after
every earlier row has started. The default pool has at least 5 threads, so the one
blocked late row leaves room. `run_job`'s `subprocess.run(timeout=600)` stays as the
hang guard. It is not a timing assumption of the test.
- The late-row case runs on async only. On the sync path, rows run in order, so a waiting
row 199 would block the second commit forever.
- No library change, no changelog fragment (test only, as #347), no design_issues entry.

## Out of scope (report in worker_done)

These are the six interrupt tests in `tests/test_pipeline/test_job_run_in_event_loop.py`
(`test_interrupt_while_running_inside_a_loop_stops_the_run`,
`test_second_interrupt_returns_and_a_late_function_writes_nothing[*]` and
`test_second_interrupt_names_an_async_function_blocking_the_loop[*]`). Their time bounds:
- `threading.Timer(1.0, _thread.interrupt_main)`: the second press assumes the first has
been handled and the run is waiting for the stuck function within 1 s.
- `time.sleep(0.05)` per row with `calls_when_stopped < N_SLOW_ROWS`, then a
`time.sleep(0.2)` quiet window.
- `stuck_release.wait(timeout=30)` and the 30 s deadline loops are hang guards only.

Making them event-driven needs an observable "run is waiting for the stuck function" state
from the library. That is not a few lines, so it is not in this PR.

### Task 1: Failing regression case (late row)

**Files:**
- Modify: `tests/test_pipeline/test_job_crash_recovery.py`

- [ ] Add a seventh script argument `late` ("none" or a row number). In `double`, when
`crash_kind == "fn"` and `x == late`, wait on `earlier_batches_committed`. Add
`late: int | None = None` to `run_job`, passed as `str(late)` or `"none"`.
- [ ] Parametrize the crash test over `sync`, `async`, `async-late-row` (async, late 199)
with the old assertions. Run it: `async-late-row` fails at `min(calls) >= 200`.
- [ ] WIP commit.

### Task 2: Order-free assertions, unbounded wait

- [ ] Add `stored_rows(store, kind) -> set[int]`: `k` of the pipeline table, `y // 2` of
the result table.
- [ ] The crash test asserts:
- two whole batches: 200 pipeline records on 200 distinct rows, each with a committed
result;
- nothing at or after the crash row stored (`max(computed) < 234`);
- sync: `recorded == computed == set(range(200))`;
- late row: the late row is neither recorded nor computed;
- the rerun computes exactly the rows without a committed result:
`sorted(calls) == sorted(set(range(N_ROWS)) - computed)`;
- `stable(store) == uninterrupted`.
- [ ] Make the wait `earlier_batches_committed.wait()`, and update the script comment and
the module docstring.
- [ ] Run the file on 3.12 and 3.13. Loop: before (origin/main test, late row injected)
vs after, ≤ 50 iterations, sequential.
- [ ] ruff format/check, basedpyright, commit, full suite, push, PR (`Refs ITL-1001`), Copilot review.

## Progress (2026-11-13)

- Task 1 done: `async-late-row` failed with the old assertions (`assert False` at the
`min(calls) >= 200` line, as on CI); the other three cases passed.
- Task 2 done: 4/4 pass on 3.12 and 3.13; ruff clean; basedpyright exit 0, no new
warnings.
- Before (origin/main test, row 199 late, scratch repro): old assertion failed 20/20;
order-free check held 20/20 (records 200, results 207, rerun min 199).
- After: 25/25 sequential runs of the two async cases passed.
110 changes: 77 additions & 33 deletions tests/test_pipeline/test_job_crash_recovery.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,9 @@
subprocess, as a power loss or ``kill -9`` would) on a ``DeltaTableDatabase``:

- in the middle of a batch: exactly the committed batches are stored, and the rerun
computes only the rows that were not;
computes only the rows that were not. On the async path rows finish, and so are
recorded and committed, in any order: a batch holds the rows that finished first, not
the first rows of the input;
- between a batch's result commit and its pipeline-record commit, also in a batch that
starts with rows whose results were already stored: the results are kept, the rerun
computes nothing for them and writes their pipeline records.
Expand All @@ -19,7 +21,7 @@
import sys
import textwrap
from pathlib import Path
from typing import Any
from typing import Any, cast

import deltalake
import pyarrow as pa
Expand All @@ -30,21 +32,29 @@
N_ROWS = 250

# Runs one job over N_ROWS rows. Arguments: store path, calls file, commit_every_rows,
# orchestrator ("sync" or "async"), crash ("none", "fn:<x>" or "flush:<n>").
# "fn:<x>" exits while computing row x, once the batches of the rows before x are
# committed; "flush:<n>" exits on the n-th store-wide flush. The run's first flush
# commits what was pending before it; each later one is a node's commit, called right
# after the node committed its result table.
# orchestrator ("sync" or "async"), crash ("none", "fn:<x>" or "flush:<n>"), late row
# ("none" or a row number). "fn:<x>" exits while computing row x, once the batches of the
# rows before x are committed; "flush:<n>" exits on the n-th store-wide flush. The run's
# first flush commits what was pending before it; each later one is a node's commit,
# called right after the node committed its result table.
#
# The async orchestrator computes rows on a thread pool ahead of the stage that records
# them and commits every commit_every_rows rows, so without a wait row x could be
# computed before the batches before it are committed (ITL-1001). Row x and every row
# after it wait until the store has finished the run's first flush and one flush per
# batch before x, so the crash falls in the batch that holds x, after the earlier
# batches are committed and before the result of any row from x on is stored. On the
# sync path the rows come in order and the wait returns at once. The wait is bounded: if
# the commits never come, the job exits with code 4, which run_job rejects, so the test
# fails instead of hanging.
# sync path the rows come in order and the wait returns at once.
#
# The pool's rows also finish in any order, and the node records them in the order they
# finish. A late row (with "fn:<x>" only, async only) waits for the same commits, so it
# finishes after the batches before x are committed without it: a row of the input's
# first batches that a loaded machine finishes late, made deterministic.
#
# The wait has no time bound, because it cannot deadlock: the pool starts rows in input
# order, so rows from x on only take threads once every row before x has started, and
# all of those return except the one late row. run_job's subprocess timeout is the hang
# guard if a change ever breaks that.
JOB_SCRIPT = textwrap.dedent(
"""
import os
Expand All @@ -57,21 +67,15 @@
from orcapod.pipeline.async_orchestrator import AsyncPipelineOrchestrator
from orcapod.sources import DictSource

store_path, calls_path, every, orchestrator, crash = sys.argv[1:6]
store_path, calls_path, every, orchestrator, crash, late = sys.argv[1:7]
crash_kind, _, crash_arg = crash.partition(":")
earlier_batches_committed = threading.Event()
if crash_kind == "fn":
flushes_before_crash = 1 + int(crash_arg) // int(every)


def wait_for_earlier_batches() -> None:
if not earlier_batches_committed.wait(timeout=120):
print(
"the batches before the crash row were never committed",
file=sys.stderr,
flush=True,
)
os._exit(4)
earlier_batches_committed.wait()


@function_pod(output_keys="y")
Expand All @@ -82,6 +86,8 @@ def double(x: int) -> int:
wait_for_earlier_batches()
if x == int(crash_arg):
os._exit(3)
if crash_kind == "fn" and late != "none" and x == int(late):
wait_for_earlier_batches()
return 2 * x


Expand Down Expand Up @@ -125,7 +131,12 @@ def __getattr__(self, name: str) -> Any:


def run_job(
tmp_path: Path, store: Path, every: int, orchestrator: str, crash: str = "none"
tmp_path: Path,
store: Path,
every: int,
orchestrator: str,
crash: str = "none",
late: int | None = None,
) -> tuple[int, list[int]]:
"""Run the job in a subprocess; return its exit code and the rows it computed."""
script = tmp_path / "job.py"
Expand All @@ -134,7 +145,7 @@ def run_job(
calls.write_text("")
done = subprocess.run(
[sys.executable, str(script), str(store), str(calls), str(every)]
+ [orchestrator, crash],
+ [orchestrator, crash, "none" if late is None else str(late)],
capture_output=True,
text=True,
timeout=600,
Expand Down Expand Up @@ -165,6 +176,21 @@ def row_count(store: Path, kind: str) -> int:
return 0 if table is None else table.num_rows


def stored_rows(store: Path, kind: str) -> list[int]:
"""The input rows (``x``) of the committed rows of the ``kind`` table, sorted.

A pipeline record names its row by the tag ``k``, which equals ``x``; a result
holds ``y = 2 * x``.
"""
table = read(tables(store).get(kind))
if table is None:
return []
# Both columns are non-null int64; to_pylist is typed as possibly None.
if kind == "pipeline":
return sorted(cast("list[int]", table["k"].to_pylist()))
return sorted(y // 2 for y in cast("list[int]", table["y"].to_pylist()))


def stable(store: Path) -> dict[str, list[tuple[object, ...]]]:
"""The stored columns that an interrupted and an uninterrupted run must share.

Expand Down Expand Up @@ -205,27 +231,45 @@ def uninterrupted(tmp_path_factory: pytest.TempPathFactory) -> dict[str, list[An
return stable(tmp / "store")


@pytest.mark.parametrize("orchestrator", ["sync", "async"])
@pytest.mark.parametrize(
("orchestrator", "late"),
[("sync", None), ("async", None), ("async", 199)],
ids=["sync", "async", "async-late-row"],
)
def test_crash_mid_batch_keeps_committed_batches(
tmp_path: Path, orchestrator: str, uninterrupted: dict[str, list[Any]]
tmp_path: Path,
orchestrator: str,
late: int | None,
uninterrupted: dict[str, list[Any]],
) -> None:
"""Killed at row 234 with 100 rows per commit: 200 rows stored, 50 recomputed."""
"""Killed at row 234 with 100 rows per commit: two whole batches stored.

The rerun computes exactly the rows without a committed result. On the async path
the two batches are the 200 rows that finished first, whichever they are; the
late-row case makes row 199 finish after the second commit.
"""
store = tmp_path / "store"
code, _ = run_job(tmp_path, store, 100, orchestrator, crash="fn:234")
code, _ = run_job(tmp_path, store, 100, orchestrator, crash="fn:234", late=late)
assert code == 3
results, records = row_count(store, "result"), row_count(store, "pipeline")
recorded = stored_rows(store, "pipeline")
computed = stored_rows(store, "result")
# Two whole batches: 200 rows with a pipeline record, each with a committed result
# (results are committed before their pipeline records).
assert len(set(recorded)) == len(recorded) == 200
assert set(recorded) <= set(computed)
assert len(set(computed)) == len(computed)
# Nothing from the crash row on is stored.
assert max(computed) < 234
if orchestrator == "sync":
assert (results, records) == (200, 200)
else:
# Results are committed before their pipeline records, so results of rows
# still on their way to being recorded may be committed too.
assert records == 200 and 200 <= results < 234
assert recorded == computed == list(range(200))
if late is not None:
# The late row finished after the second commit, so neither is stored.
assert late not in computed

code, calls = run_job(tmp_path, store, 100, orchestrator)
assert code == 0
# Only rows without a committed result are computed again.
assert len(calls) == N_ROWS - results
assert min(calls) == results if orchestrator == "sync" else min(calls) >= 200
# Exactly the rows without a committed result are computed again.
assert sorted(calls) == sorted(set(range(N_ROWS)) - set(computed))
assert stable(store) == uninterrupted


Expand Down
Loading