storage-postgres: a_caller_cancelled_between_the_commits_does_not_strand_the_data failed once in CI #928

a_caller_cancelled_between_the_commits_does_not_strand_the_data in crates/storage-postgres/tests/table_snapshot.rs (line 3368 on main at e681b8b, from #395) failed once in CI and passed when the job ran again.

The failure

Run 42802597266, job run-postgres-storage-tests (129783635056), on PR #732 at 147378a. That PR's own commit does not change crates/storage-postgres. The test panicked at assert_no_orphan (line 3040):

while the detached task runs: data tables ["\"_ddb_d4a840d1-4610-4e88-a435-2c7201e4b871\"", "\"_ddb_168420e3-25b0-47cb-9dc2-9522dd6be2ab\""] left without a catalog row

The binary's result was 88 passed, 1 failed, 2 ignored. The rerun of the same job (129795950948) passed.

What it means

The test drops a DeleteTable request between the catalog commit and the data drop. It then checks that the catalog row stays while the data tables remain. Once, the check found both data tables of the table and no catalog row. So the delete path may have a window where the catalog row is gone and the data tables are still there. Or the test's pause does not always hold the detached task at the point where the check runs. This was not investigated further.

In the other Integration Tests runs of the last 40, this job passed.