Three test_fleet.py tests fail locally while passing in CI #1354

Problem

Three tests of the benchmark harness fail locally on an unmodified master checkout, while the same tests pass in the Benchmark tooling workflow of CI:

• benchmarks/tests/test_fleet.py::TestWork::test_losing_a_slot_does_not_stop_the_job
• benchmarks/tests/test_fleet.py::TestWork::test_a_failed_job_whose_quiescing_is_throttled_stops_the_worker
• benchmarks/tests/test_fleet.py::TestWork::test_a_failed_job_quiesced_after_throttling_keeps_the_worker

Each local run of these tests takes about 9 minutes of wall-clock time with about 4 seconds of CPU time, so the tests spend almost all of it waiting.
A test that passes in CI and fails, slowly, on a developer machine depends on something in its environment that the test does not control.
just benchmark test is part of the validation for changes to result-set, cursor, and filesystem internals, so these failures make every local run take about 30 minutes and hide new regressions among known ones.

One failure, from test_losing_a_slot_does_not_stop_the_job:

E       AssertionError: assert ['attempt-1',..., 'attempt-2'] == ['attempt-1', 'attempt-1']
E         Left contains one more item: 'attempt-2'

tests/test_fleet.py:1672: AssertionError
------------------------------ Captured log call -------------------------------
WARNING  pyathena_bench.fleet:fleet.py:1147 Athena throttled quiescing first; retrying

The test makes one more attempt than expected after a throttled quiesce, which suggests that a wait, deadline, or retry in the work loop depends on time or on an environment the test does not pin.

Reproduction

At c135e21 (also at 4a0cadb):

R=$(just --evaluate BENCHMARK_UV)
eval "$R pytest -p no:cacheprovider tests/test_fleet.py -k 'losing_a_slot or quiescing_is_throttled or quiesced_after_throttling'"

Results:

• 3 failed in 558 s; just benchmark test reports the same three failures (336 passed, 2 skipped).
• With AWS_EC2_METADATA_DISABLED=true, as the workflow sets it: the same three failures, 558 s.
• With AWS_CONFIG_FILE=/dev/null AWS_SHARED_CREDENTIALS_FILE=/dev/null and no AWS_* variables in the environment: test_losing_a_slot_does_not_stop_the_job alone still fails, 548 s. AWS configuration and credentials are therefore not the cause.

The Benchmark tooling workflow (.github/workflows/benchmarks.yaml, job offline, just benchmark test) passed on master and on #1353.

Environment

• WSL2 (kernel 6.18.54.1-microsoft-standard-WSL2), Linux.
• Benchmark environment from BENCHMARK_UV: Python 3.12 via uv run --directory benchmarks --locked.
• CI: ubuntu-latest, Python 3.12.

Proposed fix (optional)

First find what the tests wait for: run one test with -v --log-level=DEBUG, a faulthandler dump during the wait, or by reading the work loop around fleet.py:1147 and the fakes of the three tests.
Candidates are a wall-clock deadline or lease (time.time(), for example the lease expires that the test sets), backoff sleeps that the fakes do not replace, and the number of CPUs or the scheduling of threads.
Then make the tests pin that input, so that they pass and end quickly in both environments, without weakening their assertions.