Skip to content

Clear key-lock health after successful lease acquisition - #998

Open
zlareb1 wants to merge 10 commits into
mainfrom
fix/key-lock-health-recovery
Open

zlareb1 wants to merge 10 commits into
mainfrom
fix/key-lock-health-recovery

Conversation

@zlareb1

@zlareb1 zlareb1 commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Related: #1094

Related: #1076

Related: #1025

Changelog category

  • Bug Fix (user-visible misbehavior in a release)

Changelog entry

Health clears a previous memory key-lock failure after a later lease acquisition succeeds. Busy locks and acquisition failures continue to refuse the protected operation.

Why

The lock records backend failures in provider health but never records successful acquisition. A recovered lock therefore leaves health degraded. Record success after receiving a lease, before entering the caller's body, so an unrelated body error cannot be counted as an acquisition failure.

Scope and risk

Replicated-erasure lease provisioning and its unit regression controls are on main. This branch preserves them unchanged. No production lease or erasure rule changes.

One server health module, one Cell proof helper, one integration fixture and six unit-test modules change. The production change records success for one health entry after confirmed lease acquisition. Regression cases cover recovery, a later body error, busy acquisition, another provider's failure and lease release.

The cancellation fixture adds bounded coroutine and signal state to its existing child-stack diagnostic after the unchanged five-second failure. It excludes input, arguments, locals, task names and raw exception text. The original cancellation failure still fails. Its cause remains unknown.

The search-deadline fixture uses a private idle clock for the first-content case. Its original 400 ms budget, fake leg delays, assertions and cancellation finalizers remain. Controls exercise pre-leg elapsed-time sensitivity, genuinely blocked legs, orphan rejection and binding restoration. This fixture correction does not establish the historical scheduling cause or change production deadlines.

The composed test prerequisites scope SCIM person-query observations to their decision tag, keep Cell cleanup watchdogs on a private idle clock, and reduce repeated JSON encoder setup in Cell proof comparisons. Existing refusal controls, cleanup bounds, typed comparisons and field-mutation assertions remain intact. The local encoder-count checks do not establish end-to-end Cell performance.

Compatibility, rollout and rollback

  • Wire: no operations, fields or defaults change.
  • Schema, settings and operator steps: none.
  • Rollback: reverting restores the previous health reporting; no stored data changes.

Tests

This head includes the exact reviewed cancellation diagnostic and search-deadline fixtures. All 79 current focused cases pass across memory key-lock health, provider health, cancellation diagnostics and search deadlines. Scoped lint, formatting, the existing strict type check, whitespace and prose checks pass. Full exact-head hosted CI and a new independent whole review remain required.

Earlier feature qualification

The following feature-specific results were recorded before this composition. They are prior-source receipts, not new tests of this composed head.

  • The three erasure-fixture regressions pass after current-main composition.

  • LOOM_ENV_FILE= .venv/bin/python -m pytest tests/unit/test_memory_key_write_lock.py tests/unit/test_provider_health.py -q: 32 passed.

  • LOOM_ENV_FILE= .venv/bin/python -m pytest tests/unit/test_native_command_retirement.py -k 'prompt_cancellation or prompt_answer' -q: nine passed, five deselected.

  • Direct prerequisite modules cover 10 SCIM observation cases, 47 typed default-comparison cases and nine cleanup cases.

  • Scoped Ruff lint and format, strict mypy for the changed server module, original-test assertion comparisons and complete-diff whitespace checks pass.

  • A separate parser cost assertion failed in earlier hosted CI. Its cause remains unresolved; focused results do not qualify that failure or replace full exact-head hosted CI.

AI assistance in the code

  • AI-generated: Codex wrote the health correction and fixture diagnostics, then checked acquisition ordering, refusal paths, per-provider isolation and release after body errors.

  • I can explain every line of this diff and how it interacts with the rest of Loom.

Checklist

  • No generated interfaces, migrations, plugins or frontend files change.
  • No operator or wire changes require documentation updates.
  • Commits are signed off.
  • Full exact-head GitHub CI passes.

@zlareb1
zlareb1 force-pushed the fix/key-lock-health-recovery branch from 08ab7b8 to 1055d3c Compare November 20, 2026 08:20
@zlareb1
zlareb1 force-pushed the fix/key-lock-health-recovery branch from 1055d3c to 30ee114 Compare November 20, 2026 12:49
@vercel

vercel Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
loom-docs Building Building Preview Nov 21, 2026 12:53am UTC

Request Review

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 772e2c3)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit e3e9b10)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit c74ea94)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Shared client schema reads run with an independent context. Record only decision-tagged statements while retaining unbounded decision reads as failures.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 57327d5)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Exercise each default-field refusal and JSON numeric distinction before consolidating repeated default comparisons.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 73ef63b)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit cb594d0)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep every default-field and JSON type check while avoiding a separate encoder setup for each field.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 0bfc712)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit a94d2f9)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep the blocked close, deadline observers and loop timers on one private
clock. A controlled independent clock advance reproduces the elapsed-time
dependency while the original cleanup and refusal assertions stay intact.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 4b03db4)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 0796d4d)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
The fake leg delays and request budget share a private event loop clock. Preserve the original timing inputs and assertions, and require blocked legs to time out and cancel.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit bfb9d40)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@zlareb1
zlareb1 force-pushed the fix/key-lock-health-recovery branch from 84d6257 to 58b9de6 Compare November 21, 2026 00:53

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant