Skip to content

Recheck the resource census after failed RSS reads - #1018

Draft
zlareb1 wants to merge 6 commits into
mainfrom
fix/cell-rss-census
Draft

zlareb1 wants to merge 6 commits into
mainfrom
fix/cell-rss-census

Conversation

@zlareb1

@zlareb1 zlareb1 commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Related: #995

Changelog category (leave one):

  • CI Fix or Improvement

Changelog entry

The Cell resource census repeats a complete process-tree observation after a verified descendant disappears during an RSS read. A failed status-file read cannot count as zero memory while the process identity remains present.

Why

A process can exit after its identity check and before its RSS status is read. ProcessLookupError from that read escapes the existing disappearance handler. FileNotFoundError becomes zero RSS, which can admit the sample while the identity remains present. Hosted regression tests reproduce both behaviors before this fix.

Scope and risk

Move the unchanged RSS read and limit check into the existing child-census handler. A fresh identity check must verify that a nonroot disappeared before the entire incomplete traversal is discarded. A fresh complete forest is required to renew the sample. Root loss, a present process whose status read failed, PID reuse, other I/O errors, excess RSS and stale observations refuse admission.

The existing three-attempt, three-second bound and all resource, workload, proof and statistical limits remain unchanged. The existing zero fallback after a successful status read with no VmRSS field is also unchanged; that separate parsing edge is outside this fix. Process cleanup and signaling are unchanged.

The independently landed placement-test repair is retained exactly from the adopted parent. Its private asynchronous clock, prepared inert transport, six cases, numeric limits and task-leak assertion remain. This PR now changes only the RSS reader, forest census and their resource tests. No production placement code changes here.

Compatibility, rollout and rollback

  • Wire changes: none.
  • Operator steps: none.
  • The change affects the acceptance fixture and has no data or schema effects. Rollback restores the earlier census behavior.

Tests

  • On the test-only source, the five new hosted cases produced the predicted two failures and three passes. The failures are a present identity admitted after FileNotFoundError and a vanished descendant refused after ProcessLookupError. Completed CI job.
  • The original 21 cases remain unchanged. Fifteen additional cases cover those read races, readable status, root loss, other errors, a replacement child's RSS, the unchanged deadline and the three-forest bound. All 36 cases passed on the preceding fixed component head. RSS regression job. The sibling placement repair also passed all six cases. Placement regression job. These component results remain source-bound historical evidence; final-head full CI remains required.
  • On an earlier combined head, all fifty other checks passed but Cell acceptance failed at its unchanged measurement deadline. Nine arms completed and the last arm was interrupted; the workload and provenance were incomplete, so no performance or acceptance pass is claimed. That failure did not show an RSS exception or reproduce the fixed RSS race. Its exact timing cause remains unknown.
  • The current parent adoption preserves all three RSS source and test images exactly and retains the landed shared placement fixture. All thirty-six directly related resource tests pass in a bounded focused local check; scoped Ruff lint and formatting checks pass for the three concern files. The earlier local admission refusal remains recorded. These small checks do not qualify full CI. A fresh independent entire-head review and all required hosted gates must pass before merge.

The preceding published head failed the two replicated-erasure integration cases because their isolated database lacked _migration_lock; fenced memory writes could not read the lease holder. This head includes the landed fixture repair. All three RSS source and test images remain byte-identical to the preceding head. The failed cohort remains recorded. Fresh independent entire pushed-source review and all required hosted gates are pending for this composition; earlier results do not qualify it.

AI assistance in the code

  • AI-generated: Codex prepared the diff. Independent source reviews and hosted regressions check the identity, full-forest and refusal requirements.

  • I can explain every line of this diff and how it interacts with the rest of Loom.

Checklist

  • Hosted regression RED confirmed the predicted failures before the source fix.
  • Final full CI and a fresh independent whole-change review pass before merge.
  • No generated outputs, migration, plugin version or frontend change is required.
  • Commits are signed off.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@zlareb1 zlareb1 changed the title Test RSS census exit and unreadability races Recheck the resource census after failed RSS reads Nov 20, 2026
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@zlareb1 zlareb1 mentioned this pull request Nov 20, 2026
4 of 5 tasks
Preserve the RSS read-race fix and all resource tests while composing the independently landed shared HTTP fixture repair.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@vercel

vercel Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
loom-docs Ready Ready Preview Nov 20, 2026 11:49pm UTC

Request Review

The integration fixture provisions the lease table used by fenced memory clearing.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant