Skip to content

Drive trigger cleanup checks with one private clock - #1095

Open
zlareb1 wants to merge 3 commits into
mainfrom
fix/cell-attach-trigger-deadline-clock
Open

zlareb1 wants to merge 3 commits into
mainfrom
fix/cell-attach-trigger-deadline-clock

Conversation

@zlareb1

@zlareb1 zlareb1 commented Nov 21, 2026 •

Copy link
Copy Markdown
Member

Related: #1094

Changelog category (leave one):

  • CI Fix or Improvement (changelog entry is not required)

Changelog entry

Not required.

Why

The trigger-try unit fixture gives work a short event-loop timeout and freezes its cleanup clock. Its outer watchdog also uses the event-loop clock. Scheduling can consume the watchdog before the work timeout and cleanup finish their separate turns. A controlled 250 ms loop-clock advance reproduces the reset case's stuck-task assertion; its zero-jump neighbour passes. The historical CI scheduling cause remains unknown.

The six stall cases explicitly request the same function-scoped loop as their clock fixture. A runtime identity assertion checks that the test and fixture use that loop before setting phase clocks. The cases share a private idle clock between the event loop, owner phase and Postgres close. Work and cleanup still run their real cancellation code. The watchdog, phase bound and all original cancellation rounds remain unchanged. This fixture does not qualify real wall-clock latency.

Scope and risk

One existing unit module changes. Every original test body, all six stall parameters, the 200 ms watchdog and three 10 ms cancellation rounds remain unchanged. The clock fixture activates only for this parameterized case and its new controls.

A genuinely blocked public close must still time out, terminate its fake connection and cancel its pending close task. An orphan-task control requires the fixture to cancel, drain and reject newly created tasks on its private function-scoped loop. Scoped patches restore all clock and helper bindings. The fixture relies on the selector loop used by the unit test platform.

Compatibility, rollout and rollback

  • Wire: none.
  • Operator steps: none.
  • Rollback: revert the test-only change; no stored data or generated files change.

Prior-source regression evidence

  • Controlled scheduling regression: one failure and one passing neighbour before the fixture; both controls pass with the fixture.
  • Collected loop-identity regression: all six original parameter cases fail before the explicit function-scoped test marker; the identical command passes all six after it.
  • .venv/bin/python -m pytest tests/unit/test_cell_attach.py -q -k 'a_trigger_try_deadline_covers_transaction_and_lock_cleanup or trigger_try_clock' --override-ini=addopts=: 10 passed, including all six original stall cases.
  • Ruff lint and format checks, public body and added fixture text prose checks, and git diff --check pass.
  • AST comparison preserves all 52 original test bodies and all 119 original assertions.
  • Full hosted CI and a fresh independent whole-diff review remain required.

Tests

All three new clock-control functions explicitly request function-scoped loops, matching the six collected original stalls. A separate temporary collection check rejects the four collected control cases before the marker correction, then accepts all ten selected cases afterward. The same ten cases pass execution. Every published and original test body, all original assertions, phase and watchdog deadlines remain unchanged. Full-module prose remains failed on the same three baseline literals; added text and this body pass their scoped prose checks.

On this current-main composition, all ten directly related original stall and clock control cases pass. Scoped Ruff lint and formatting, added-text prose, public-body prose and committed-range whitespace checks pass. All 55 published test bodies retain identical ASTs and the reviewed loop marker and identity guard match their exact source commit. The full-module prose checker has three unchanged baseline test literals; the preserved baseline equivalence receipt records those failures. Full hosted CI and a fresh independent whole review remain required.

AI assistance in the code

  • AI-generated: Codex wrote the fixture and controls. The original test bodies, deadlines, production source and focused results were checked.

  • I can explain every line of this diff and how it interacts with the rest of Loom.

Checklist

  • Full required hosted CI passes; local checks are limited to this concern.
  • Generated files: not applicable.
  • Migrations: not applicable.
  • Shipped plugin files and versions: not applicable.
  • Frontend: not applicable.
  • Docs and operator steps: not applicable.
  • Repository text contains no private operational details.
  • Commits are signed off.

@vercel

vercel Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
loom-docs Ready Ready Preview Nov 21, 2026 1:50am UTC

Request Review

@zlareb1
zlareb1 force-pushed the fix/cell-attach-trigger-deadline-clock branch from f6ab8fd to 559c141 Compare November 21, 2026 01:32
The work watchdog and cleanup clocks share one function-scoped event loop. Preserve all original stall cases and bounds, and require a genuinely blocked close to time out and cancel.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
The test marker uses the same function-scoped loop as the idle clock fixture. Assert their runtime loop identity before setting phase clocks so scope drift fails each collected stall case.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 9f7988578ecc0654720996156bff919ae33199e5)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@zlareb1
zlareb1 force-pushed the fix/cell-attach-trigger-deadline-clock branch from 559c141 to 8c5c767 Compare November 21, 2026 01:47

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant