Skip to content

Recheck renewed supervisor leases after a wait expires - #1002

Open
zlareb1 wants to merge 5 commits into
mainfrom
fix/supervisor-renewal-timeout-race
Open

zlareb1 wants to merge 5 commits into
mainfrom
fix/supervisor-renewal-timeout-race

Conversation

@zlareb1

@zlareb1 zlareb1 commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Related: #1041
Related: #1072
Related: #1077
Related: #1087

Changelog category

  • Bug Fix (user-visible misbehavior in a release)

Changelog entry

The worker supervisor rechecks lease freshness after a wait times out, so a renewal that completed during the wait keeps its workers running.

Why

A renewal can complete while the wait for the previous deadline resumes. Recheck the lease's current freshness after an empty wait before deciding to stop workers.

Scope and risk

The supervisor continues waiting when the lease is fresh. Missing and expired leases keep the existing stop path. Regression controls cover a renewal during the wait and the missing and expired cases. Lease duration, renewal cadence, cancellation and stop behavior are retained.

The qualification composition includes related Cell proof encoder setup, SCIM query-recording controls, replicated-erasure fixture regressions and cleanup-clock controls. The erasure fixture already on main remains unchanged. All original cleanup deadlines and assertions are retained. Cell workload, proof pages, resource census, global deadlines and latency and throughput bounds remain unchanged. Complete acceptance, latency and throughput require hosted qualification on this composition.

Compatibility, rollout and rollback

  • Wire, schema, settings, defaults and operator steps: none.
  • Rollback: revert the freshness recheck and qualification changes. No persisted data changes.

Tests

This head matches the exact three-way composition of the preceding qualified source and current main. The erasure fixture and its canonical unit regression module are inherited from main. On this composed head, all three erasure unit cases, scoped Ruff, Python formatting, committed-range whitespace and public-body prose checks pass. The existing strict type check for the changed production module passes. Full required hosted CI and a fresh independent whole review remain required for this exact head.

env LOOM_ENV_FILE= .venv/bin/python -m pytest tests/unit/test_erasure_replicated_fixture.py -q

Earlier feature qualification

The following feature-specific results were recorded before this composition. They are prior-source receipts, not new tests of this composed head.

  • LOOM_ENV_FILE= .venv/bin/python -m pytest -q tests/unit/test_cell_trigger_default_comparison.py tests/unit/test_scim_person_query_observation.py tests/unit/test_erasure_replicated_fixture.py tests/unit/test_cell_attach_cleanup.py tests/unit/test_supervisor_lease.py: 125 passed in 4.51 seconds.
  • Scoped Ruff lint and format, strict mypy for the supervisor module, original source and test-body parity, and diff whitespace checks pass.
  • Full exact-head hosted CI, including Cell acceptance and real replicated-erasure behavior, must pass before merge.

AI assistance in the code

  • AI-generated: Codex wrote the recheck and controls, then checked fresh, missing and expired leases, cancellation and unchanged failure bounds.

  • I can explain every line of this diff and how it interacts with the rest of Loom.

Checklist

  • Existing lease, workload and deadline bounds are retained.
  • No generated interfaces, migrations or operator settings change.
  • Commits are signed off.
  • Full exact-head GitHub CI passes.

@zlareb1
zlareb1 force-pushed the fix/supervisor-renewal-timeout-race branch 2 times, most recently from 0184f15 to 1f01367 Compare November 20, 2026 20:36
@vercel

vercel Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
loom-docs Ready Ready Preview Nov 21, 2026 12:10am UTC

Request Review

@zlareb1
zlareb1 force-pushed the fix/supervisor-renewal-timeout-race branch from 1f01367 to 05a5528 Compare November 20, 2026 23:07
A confirmed renewal can extend the deadline while the previous wait resumes. Use the current freshness before stopping workers, and retain genuine expiry controls.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Exercise each default-field refusal and JSON numeric distinction before consolidating repeated default comparisons.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 73ef63b)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep every default-field and JSON type check while avoiding a separate encoder setup for each field.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 0bfc712)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Shared client schema reads run with an independent context. Record only decision-tagged statements while retaining unbounded decision reads as failures.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 57327d5)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep the blocked close, deadline observers and loop timers on one private
clock. A controlled independent clock advance reproduces the elapsed-time
dependency while the original cleanup and refusal assertions stay intact.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 4b03db4)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@zlareb1
zlareb1 force-pushed the fix/supervisor-renewal-timeout-race branch from 05a5528 to 2bd301c Compare November 21, 2026 00:08

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant