Repository navigation
test: fix two start-order and teardown races in native/local tests - #7678
dwsmith1983 wants to merge 1 commit into
Conversation
partition_order_runs_at_most_the_given_number_of_partitions compared the start order of two partitions that start concurrently. Check the started set through assert_executed_once instead, which sorts. cancel_wakes_a_waiting_reader asserted the plan was released as soon as the cancelled reader returned, but an aborted partition task that has not run yet still owns its start closure until a runtime worker drops it. Wait for the count to reach zero under a deadline, as the other teardown checks in the crate do. Closes apache#7676 Closes apache#7677
|
@andygrove @mbutrovich @sunchao this fixes the two flaky |
sunchao
left a comment
There was a problem hiding this comment.
Reviewed 8e3be5b44c1bd671e1e079796d1a92c428d5acf0 with five independent review passes. No actionable findings.
The cancellation test still checks prompt reader wakeup and eventual graph release, while allowing Tokio to finish aborting task-owned state. Reusing assert_executed_once preserves exact partition membership and duplicate detection without requiring task start order. Both changes match the production contracts and leave runtime behavior unchanged.
Validation: source and test inspection only, with no local test execution. The exact-head fork CI run passed, including both changed tests. Upstream workflows remain action_required. Apache 1.2.0 predates native/local, so this patch changes unreleased tests and no released API.
|
Thanks @dwsmith1983. Took the liberty of moving one of your others to draft and marked this ready for review. Appreciate the help with this! |
Which issue does this PR close?
Closes #7676.
Closes #7677.
Rationale for this change
Two tests in
native/local, both added in #6883, fail intermittently and have been evicting unrelated pull requests from the merge queue (#7556, #7464, #7508). Each one asserts something the code doesn't guarantee.What changes are included in this PR?
partition_order_runs_at_most_the_given_number_of_partitionscompared the start order of two partitions that start concurrently on separate workers. It now checks the started set throughassert_executed_once, which sorts, like thewait_startedcheck a few lines above it.cancel_wakes_a_waiting_readerasserted the plan was released as soon as the cancelled reader returned. An aborted partition task that hasn't run yet still owns its start closure, and with it the plan, until a runtime worker drops the task. The test now waits up to 5 seconds for the count to reach zero, as the other teardown checks in the crate do.cancel_releases_the_graph_while_every_worker_is_busyintests/bridge.rsstill covers the stream being dropped on the calling thread.How are these changes tested?
Both are test changes. I forced each race and confirmed the old assertion failed every time with the same message as CI, while the new one passed:
With no delay planted, fmt, clippy and the full
datafusion-comet-localtest suite pass, and each test passed 200 runs in a row.