Skip to content

test: fix two start-order and teardown races in native/local tests - #7678

Open
dwsmith1983 wants to merge 1 commit into
apache:mainfrom
dwsmith1983:test/native-local-flakes
Open

dwsmith1983 wants to merge 1 commit into
apache:mainfrom
dwsmith1983:test/native-local-flakes

Conversation

@dwsmith1983

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #7676.
Closes #7677.

Rationale for this change

Two tests in native/local, both added in #6883, fail intermittently and have been evicting unrelated pull requests from the merge queue (#7556, #7464, #7508). Each one asserts something the code doesn't guarantee.

What changes are included in this PR?

  • partition_order_runs_at_most_the_given_number_of_partitions compared the start order of two partitions that start concurrently on separate workers. It now checks the started set through assert_executed_once, which sorts, like the wait_started check a few lines above it.
  • cancel_wakes_a_waiting_reader asserted the plan was released as soon as the cancelled reader returned. An aborted partition task that hasn't run yet still owns its start closure, and with it the plan, until a runtime worker drops the task. The test now waits up to 5 seconds for the count to reach zero, as the other teardown checks in the crate do. cancel_releases_the_graph_while_every_worker_is_busy in tests/bridge.rs still covers the stream being dropped on the calling thread.

How are these changes tested?

Both are test changes. I forced each race and confirmed the old assertion failed every time with the same message as CI, while the new one passed:

  • a 200 ms delay before partition 0 records its start;
  • the single runtime worker held busy for 500 ms by an unrelated task.

With no delay planted, fmt, clippy and the full datafusion-comet-local test suite pass, and each test passed 200 runs in a row.

partition_order_runs_at_most_the_given_number_of_partitions compared the
start order of two partitions that start concurrently. Check the started
set through assert_executed_once instead, which sorts.

cancel_wakes_a_waiting_reader asserted the plan was released as soon as
the cancelled reader returned, but an aborted partition task that has not
run yet still owns its start closure until a runtime worker drops it.
Wait for the count to reach zero under a deadline, as the other teardown
checks in the crate do.

Closes apache#7676
Closes apache#7677
@dwsmith1983

Copy link
Copy Markdown
Contributor Author

@andygrove @mbutrovich @sunchao this fixes the two flaky native/local tests that have been evicting PRs from the merge queue (#7676, #7677). The changes are test-only, 12 lines. It's a draft because I'm at my open PR limit, but if one of you would like to fast-track it to stop the evictions, I'll mark it ready.

@sunchao sunchao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 8e3be5b44c1bd671e1e079796d1a92c428d5acf0 with five independent review passes. No actionable findings.

The cancellation test still checks prompt reader wakeup and eventual graph release, while allowing Tokio to finish aborting task-owned state. Reusing assert_executed_once preserves exact partition membership and duplicate detection without requiring task start order. Both changes match the production contracts and leave runtime behavior unchanged.

Validation: source and test inspection only, with no local test execution. The exact-head fork CI run passed, including both changed tests. Upstream workflows remain action_required. Apache 1.2.0 predates native/local, so this patch changes unreleased tests and no released API.

@github-actions github-actions Bot added enhancement New feature or request test Testing related labels Nov 19, 2026
@andygrove
andygrove marked this pull request as ready for review November 19, 2026 22:27
@andygrove
andygrove enabled auto-merge November 19, 2026 22:27
@andygrove

Copy link
Copy Markdown
Member

Thanks @dwsmith1983. Took the liberty of moving one of your others to draft and marked this ready for review. Appreciate the help with this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request test Testing related

Projects

None yet

3 participants