Skip to content

Zombie witness starts without inherited exporter threads - #1046

Closed
zlareb1 wants to merge 1 commit into
mainfrom
fix/ci-zombie-witness-cycle-20261120-1240
Closed

zlareb1 wants to merge 1 commit into
mainfrom
fix/ci-zombie-witness-cycle-20261120-1240

Conversation

@zlareb1

@zlareb1 zlareb1 commented Nov 20, 2026

Copy link
Copy Markdown
Member

Related: #986

Changelog category (leave one):

  • CI Fix or Improvement (changelog entry is not required)

Changelog entry (one or two sentences a user of Loom can read without the diff; the release pull request collects these into the release notes):

Not required.

Why

The single-thread zombie witness inherits tracing exporter fork hooks. The failed unit job observes two live threads instead of one. An active batch exporter restarts its worker after fork, so the live witness can have two threads when the test expects one. A controlled 50 ms pause in the parent reproduces the CI assertion on the CI merge commit, its first parent and main; the test and process-proof source blobs match across those commits.

Start an isolated Python child and wait for its ready byte before reading its process identity. Exec removes inherited worker threads and fork hooks from the witness. Its pipe still controls exit, so the test observes the zombie before collecting it.

Scope and risk

Only the single-thread witness test changes. It retains live-process refusal, namespace mismatch refusal, exact single-thread zombie acceptance and acceptance after collecting the child. Cleanup remains bounded and checks successful child exit. The separate test for a zombie leader with live sibling threads keeps its assertions.

Compatibility, rollout and rollback

  • Wire: none.
  • Operator steps: none.
  • Rollback: revert the test change; no stored-data change.

Tests

  • The controlled exporter/fork probe fails with the exact CI assertion before the edit on clean main, then passes after it. The failed CI merge commit and its first parent carry identical test and process-proof source blobs.
  • pytest tests/unit/test_provider_zombie_exit.py -q: 2 passed on Linux. The adjacent checkout-byte and release-step checks, plus these cases: 14 passed.
  • make format, make format-check, make lint and make typecheck: pass.
  • PYTEST_XDIST_AUTO_NUM_WORKERS=2 LOOM_ENV_FILE= make test-unit on Linux with Python 3.12: 45,868 passed, 127 skipped, 1 expected failure and 34 subtests passed; all 251 offline harness tests passed. The existing skips and expected failure are unchanged.

The runner uses the release-matched ClickHouse local binary and checksum-verified jq 1.8.1. Its private Git metadata has the container user's ownership; Git's ownership checks remain enabled. No production code or assertion changes are needed for those runner prerequisites.

The controlled reproduction starts a batch exporter and pauses only the parent after fork:

import os
import time
import pytest
from opentelemetry.sdk.trace.export import BatchSpanProcessor, SpanExporter, SpanExportResult

class Exporter(SpanExporter):
    def export(self, spans):
        return SpanExportResult.SUCCESS

processor = BatchSpanProcessor(Exporter(), schedule_delay_millis=60000)
fork = os.fork

def delayed_fork():
    pid = fork()
    if pid:
        time.sleep(0.05)
    return pid

os.fork = delayed_fork
try:
    code = pytest.main([
        "-q", "-p", "no:randomly",
        "tests/unit/test_provider_zombie_exit.py::test_an_exact_single_thread_zombie_is_proven_stopped_before_wait",
    ])
finally:
    processor.shutdown()
raise SystemExit(code)

AI assistance in the code (leave one; writing this description with AI is fine):

  • AI-generated (an agent wrote most of the diff): Codex. Checked process ownership, readiness, zombie observation before wait, bounded cleanup and the preserved proof assertions.

  • I can explain every line of this diff and how it interacts with the rest of Loom.

Checklist

  • Format, lint, type and complete unit and harness checks pass.
  • Generated files, migrations, plugin files and frontend: none affected.
  • Operator steps and wire changes: none.
  • Public text contains no deployment details.
  • Commits are signed off.

An active tracing exporter restarts its worker after fork, so a raw-fork
witness can violate the single-thread precondition. Start an isolated
interpreter and wait for readiness before checking the process proof.

Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
@vercel

vercel Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
loom-docs Ready Ready Preview Nov 20, 2026 1:37pm UTC

Request Review

zlareb1 added a commit that referenced this pull request Nov 20, 2026
Install the matching stable SDKs and the declared native peer. Validation fixtures establish a post-exec witness and completed backend termination before checking successor ownership.

Reuses the complete published witness repair from 9ffae96 (#1046).

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
zlareb1 added a commit that referenced this pull request Nov 20, 2026
* Pin retriever installation to available SDK packages

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

* LangChain examples select published SDK artifacts

Install the matching stable SDKs and the declared native peer. Validation fixtures establish a post-exec witness and completed backend termination before checking successor ownership.

Reuses the complete published witness repair from 9ffae96 (#1046).

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

---------

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@zlareb1 zlareb1 closed this Nov 20, 2026
zlareb1 added a commit that referenced this pull request Nov 20, 2026
* Close preparation spends the shared session-end deadline

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

* Drive placement deadline tests with a private clock

Use one idle-driven clock for the mock exchange timers and owned close. Keep the six cases and their bounds, with a private-loop task census at teardown.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit e1f47e6)

* Drive directory cleanup checks with one private clock

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

* Docs: serve documentation from the docs subpath (#1042)

* Serve documentation from the docs subpath

The documentation build keeps its links, assets, and discovery files below the configured path.

Signed-off-by: Shaun Struwig <41984034+Blargian@users.noreply.github.com>

* Trigger documentation preview

Signed-off-by: Shaun Struwig <41984034+Blargian@users.noreply.github.com>

* Redirect documentation preview root

Preview deployments open the documentation path when their root URL is requested.

Signed-off-by: Shaun Struwig <41984034+Blargian@users.noreply.github.com>

* Placement deadline tests use one simulated clock

Work deadlines, cleanup, and the elapsed-time assertion use one private clock.
The selector advances it when the loop waits, so scheduler pauses cannot expire
a simulated reply or consume its cleanup allowance. Keep all six cases and
their duration bounds, and fail teardown if the private loop retains a task.

Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>

---------

Signed-off-by: Shaun Struwig <41984034+Blargian@users.noreply.github.com>
Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
Co-authored-by: Lareb Zafar <lareb.zafar@clickhouse.com>

* DSPy retrievers over Loom knowledge and memory search (#1049)

* memory.ask matches bridge slugs literally (#1037)

* memory.ask matches bridge slugs literally

Escape LIKE metacharacters so slug underscores cannot select another subject.

Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>

* Expect literal bridge slugs in the governed-conflict test

The namespace regression must assert the escaped pattern used by the subject filter.

Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>

---------

Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>

* Record worker context after linear-cost assertions fail (#1058)

Selected durations alone omit worker allocation and collection state. Attach numeric context after the original assertion fails so the cost calls and decision remain unchanged.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

* Document memory selection for coding agents and software factories (#1021)

* Document memory selection for coding agents and software factories

Give the existing use-case pages distinct search titles and concrete recall, isolation and evidence-handoff checks without claiming comparative results.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

* Format coding-agent and factory memory selection tables

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

---------

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

* Document memory write and worker progress regression tests (#1043)

* Document memory write and worker progress regression tests

Exercise derived effects, stored cursors and complete SQL rows so regression checks observe the boundaries they claim to protect.

Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>

* Test warm worker passes beside cursor restarts

A restart can recover a missed boundary event through cold reconciliation, so the guide must retain warm continuation regressions.

Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>

* Concurrency tests cover changed content at an equal version

Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>

* Document committed-write cleanup and private cache regression tests

Cancellation during fence release must not skip effects owed by an acknowledged
write. Caches of private strings need retention and erasure coverage as well as
result checks.

Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>

---------

Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>

* Framework examples select matching SDK artifacts (#1064)

* Framework examples select matching SDK artifacts

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

* Framework examples install matching published SDKs

Select available stable artifacts and their native peers, keeping DSPy on its matching checkout. Give the invalid-identity fixture its own product clock so semantic closure assertions do not spend real phase time. Preserve all real deadline and cancellation cases.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

---------

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

* Pin retriever installation to available SDK packages (#1061)

* Pin retriever installation to available SDK packages

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

* LangChain examples select published SDK artifacts

Install the matching stable SDKs and the declared native peer. Validation fixtures establish a post-exec witness and completed backend termination before checking successor ownership.

Reuses the complete published witness repair from 9ffae96 (#1046).

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

---------

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

* Cell checks share topology and catalog helpers (#906)

Related: #776

### Changelog category

- Improvement

### Changelog entry

Cell checks share topology, catalog and synchronization helpers across command and server code.

### Why

Complete cell reads depend on topology validation, the caller's effective privileges, replica identity and synchronization. These implementations live in CLI modules, which server code cannot import. Shared owners let both layers use the same checks without copying them.

### Scope and risk

The complete topology module moves to `server/core/cell_ch.py`. Catalog, grant, default-role, row-policy and replica checks move to `server/insights/store/cell_catalog.py`; synchronization and budget helpers move to `server/core/cell_sync.py`. Callers and test patch points use those owners, and the old topology module is deleted.

SQL, query settings and tags, defaults, refusal conditions, exception identity, row interpretation and ordinary update behavior remain unchanged. A missed caller import or patch point is the main risk. Bounded participant and observation joins remain unimplemented; this extraction adds no later-cell activation or readiness result.

The configuration and placement foundation is merged. Its raw connection ownership, retirement admission, typed configuration refusal, deadlines and cleanup remain intact when the shared imports compose with it.

### Compatibility, rollout and rollback

- Wire and defaults: unchanged; no operations, arguments, settings, tables or stored encodings are added.
- Operator steps: none.
- Rollback: reverting restores the former internal owners and callers; there is no stored data transition.
- The English architecture page names the shared topology owner. Existing translated code references await the normal translation workflow; translated files are unchanged.

### Tests

- Static bindings retain all 37 moved definitions, all 574 audited test assertions and the single unique moved topology assertion. The merged foundation's 577 assertions in the audited test modules remain in source order, with only the sixteen existing shared-owner reference substitutions. Production assertion relocations retain their conditions and multiplicity. The older reported total counted the moved topology assertion in both source images; no assertion was removed.
- The local attempt to run sixteen directly related pure selectors is refused before a test child starts because admission detects an unknown test process. No current test pass is claimed. Source and Git state match before and after the refusal. The same selectors historically pass as 34 parameterized cases on the earlier source, covering complete replica union, topology and synchronization refusal, privileges, caller identity, row policies, extra replicas, streaming failures and budget behavior. Those passes retain their original source attribution.
- Current scoped Ruff checks and formatting checks pass for all 25 changed Python files, and the concern diff passes whitespace checks. The earlier focused attempt lacking its complete source snapshot remains invalid; its passing output is not used as evidence. Historical focused passes retain their original source attribution.
- The actual merged-parent adoption resolves one import conflict while retaining both shared refusal owners and the existing retirement implementation. Static comparison with the later target finds no concern-path overlap or new production coupling; unrelated target advancement alone does not require a source refresh.
- Final-head hosted CI reports 54 successful checks and two expected documentation publication skips. Both Postgres versions complete the native, standby, authenticated-isolation and ordered-reconciliation phases. Mixed-version checks pass. Cell acceptance passes the unchanged latency and throughput requirements, with provenance and cleanup receipts. Fresh independent entire pushed-source review reports no findings or nits. These results apply to this extraction; bounded census joins remain separate work.

### AI assistance in the code

- AI-generated: Codex wrote this change. Independent final-source review and required hosted gates remain mandatory.

- [x] I can explain every line of this diff and how it interacts with the rest of Loom.

### Checklist

- [x] Scoped static checks pass; the refused local test attempt is recorded as unrun.
- [x] Required final-head GitHub CI and fresh whole-source review pass.
- [x] Documentation names the shared owner; generated translations are unchanged.
- [x] No migration or stored data change is introduced.
- [x] Commits are signed off.

* Live MCP results protect the SDK before automatic capture (#1067)

Compose rendering and transport observation so the privacy marker reaches protection before the next automatic write.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

* Drive directory cleanup checks with one private clock (#1054)

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

---------

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: Shaun Struwig <41984034+Blargian@users.noreply.github.com>
Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
Co-authored-by: Shaun Struwig <41984034+Blargian@users.noreply.github.com>
Co-authored-by: Rahul <132360698+rahulsjha@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant