feat(runtime): Implement pull based arroyo (hackweek) - #560

Draft

tryangul wants to merge 34 commits into

mainfrom

rbroughan/pull-based-arroyo-part-2

Commits on Aug 31, 2026

Full stage based implementation of pull streaming
tryangul
committed
319e885
Handle N:1 and N:0 use cases. Refactor StageResult, handlers and organize for clarity.
tryangul
committed
da3d724
Support concurrent stage application.
tryangul
committed
da211d7
Actually commit offsets.
tryangul
committed
448488a
Add graceful termination aka cancellation support.
tryangul
committed
ada8559
Comment tweaks.
tryangul
committed
7acfc20
handle fatal kafka errors, preserve dlq headers, init coarsetime updater, move offset tracker.
tryangul
committed
a855d06
Basic rebalance handling.
tryangul
committed
e65d4ac
Factor out PipelineRunner. Update examples.
tryangul
committed
5a36b93
Formatting and comment cleanup.
tryangul
committed
543b7df
Fix stage name lifetimes.
tryangul
committed
6a6d6b9
Use singleton updater to prevent leak.
tryangul
committed
44387cd
Synchronize drain on rebalance.
tryangul
committed
cf646b1
Drain regardless of outcome
tryangul
committed
3f89896
Revert drain on rebalance due to potential threading deadlock.
tryangul
committed
2e3d449
Add BatchStage
tryangul
committed
ad8cbcc
Break out stream collector.
tryangul
committed
840632a
Add flush timer
tryangul
committed
bace648
Buffer now has an Out type
tryangul
committed
36533f3
Formatting
tryangul
committed
4f3c5a7
Add Pipeline trait
tryangul
committed
bff165d
Fmt.
tryangul
committed
15e873a
Doc test import
tryangul
committed
e7af45f
Cleanup types with aliases. Convert kafka stats logging to metrics.
tryangul
committed
c2c4b09
Cargo fmt.
tryangul
committed
f79f05b
ref(pull): remove unnecessary Arc around raw payload
tryangul
committed
4b03c33
ref(pull): use coarsetime for stage durations.
tryangul
committed
4d1c367
ref(pull): use parking_lot mutex for batch stage to avoid mutex syscall.
tryangul
committed
43fd7ed
ref(pull): boxed dyn producer for dlq handler.
tryangul
committed
13a872d
ref(pull): owned streams.
tryangul
committed
498b419
ref(pull): Map_emit.
tryangul
committed
47e5e8c
ref(pull): Fix-up pipeline runner.
tryangul
committed
47e1a08
ref(pull): Add batch time metric with flush reason for parity with push.
tryangul
committed
b274934
ref(pull): Fmt.
tryangul
committed
1bf5057

Navigation Menu

Uh oh!

Uh oh!

feat(runtime): Implement pull based arroyo (hackweek) - #560

feat(runtime): Implement pull based arroyo (hackweek)#560
tryangul wants to merge 34 commits into
mainfrom
rbroughan/pull-based-arroyo-part-2

Commits on Aug 31, 2026

Full stage based implementation of pull streaming

Handle N:1 and N:0 use cases. Refactor StageResult, handlers and organize for clarity.

Support concurrent stage application.

Actually commit offsets.

Add graceful termination aka cancellation support.

Comment tweaks.

handle fatal kafka errors, preserve dlq headers, init coarsetime updater, move offset tracker.

Basic rebalance handling.

Factor out PipelineRunner. Update examples.

Formatting and comment cleanup.

Fix stage name lifetimes.

Use singleton updater to prevent leak.

Synchronize drain on rebalance.

Drain regardless of outcome

Revert drain on rebalance due to potential threading deadlock.

Add BatchStage

Break out stream collector.

Add flush timer

Buffer now has an Out type

Formatting

Add Pipeline trait

Fmt.

Doc test import

Cleanup types with aliases. Convert kafka stats logging to metrics.

Cargo fmt.

ref(pull): remove unnecessary Arc around raw payload

ref(pull): use coarsetime for stage durations.

ref(pull): use parking_lot mutex for batch stage to avoid mutex syscall.

ref(pull): boxed dyn producer for dlq handler.

ref(pull): owned streams.

ref(pull): Map_emit.

ref(pull): Fix-up pipeline runner.

ref(pull): Add batch time metric with flush reason for parity with push.

ref(pull): Fmt.