Skip to content

Retain bounded native fault metadata for unit workers - #1097

Open
zlareb1 wants to merge 6 commits into
mainfrom
diag/pytest-native-signatures
Open

zlareb1 wants to merge 6 commits into
mainfrom
diag/pytest-native-signatures

Conversation

@zlareb1

@zlareb1 zlareb1 commented Nov 21, 2026 •

Copy link
Copy Markdown
Member

Related: #1065

Changelog category (leave one):

  • CI Fix or Improvement (changelog entry is not required)

Changelog entry (one or two sentences a user of Loom can read without the diff; the release pull request collects these into the release notes):

Unit worker crashes retain bounded native fault metadata alongside their terminal status.

Why

A worker return code identifies a fatal signal but leaves its native fault site unknown. The third unit part runs each local worker under an ancestor debugger and retains native PCs, symbols, module identities and offsets. The debugger inherits the gateway streams through saved descriptors, and a fixed bootstrap restores the original interpreter arguments and environment before execution.

An ancestor debugger uses ordinary tracing permissions. Sibling attachment can require a permission grant; core files can retain credentials. This collector makes no tracing policy, capability or trust changes, and discards raw debugger output. CI restricts only the diagnostic command soft core-size limit to zero. A piped core handler, unreadable core configuration, missing prerequisite or tracing refusal stops startup without an undebugged retry.

Scope and risk

Only the third unit part opts into collection. GDB is installed as a CI test tool. Worker count, restart allowance, random seed, timeout settings, coverage, assertions and test commands remain in place. Tracing can change scheduling, so a passing run does not establish a crash repair.

Each helper sets its own parent-death signal and checks its expected parent before execution. Forced launcher death stops its debugger and worker through those kernel lifetime edges. Registration files never authorize controller signals to another process.

Collection retains at most eight signal stops, eight threads per stop and 24 frames per thread, with a byte cap on each artifact. A handled signal stop remains distinct from the terminal result. The launcher preserves ordinary exit codes, including 139, and mirrors actual fatal signals. A bounded fault-thread summary appears in the existing job log. No arguments, locals, environment values, memory bytes or core dumps are collected.

Compatibility, rollout and rollback

  • Wire: none.
  • Operator steps: none. The opt-in test mode requires Linux, GDB and a zero core-size limit and no piped core handler. It refuses unsupported gateway configurations.
  • Rollback: reverting removes the CI diagnostic mode and its test helpers; no stored application data changes.

Tests

Focused unit regressions and an explicit controlled debugger matrix cover binary transport, exact environment restoration, ordinary and fatal outcomes, handled signals, bounded artifacts, startup refusals and helper termination. A tiny xdist run compares assertion, exception, crash and JUnit outcomes with collection enabled and disabled. The first hosted mechanism run refused a nonzero core-size limit before starting any worker. The CI commands explicitly restrict their own soft core limit; the runtime guard remains active and also refuses piped core handlers, which ignore that limit. All 43 focused cases pass in 1.87s, including the stricter privacy prerequisites. The controlled matrix previously passed on Linux with Python 3.12 and GDB 15.1; The final hosted mechanism run also refuses its piped core handler before starting a worker. The PR remains unmerged because the existing runner does not satisfy the privacy prerequisites; no handler or tracing policy is changed. Local real-fault execution is refused because the host has a piped handler. Full hosted CI is required before merge.

python -m pytest -q -n 0 tests/unit/test_native_worker.py tests/unit/test_worker_status.py
python -m tests.unit.test_native_worker --mechanism

Terminal-value read errors finalize an unknown result and quit. A register-ESRCH error after a fatal stop records an incomplete observation and leaves terminal attribution to the native exit event; it never retries or infers the terminal signal.

AI assistance in the code (leave one; writing this description with AI is fine):

  • AI-generated: Codex wrote the change. Independent agents review the complete source and validation evidence.

  • I can explain every line of this diff and how it interacts with the rest of Loom.

Checklist

  • Full hosted formatting, lint, typing and tests pass before merge; local checks are limited to this change.
  • No generated interface, schema, SDK or plugin output changes.
  • No migrations, server behavior, frontend or provider changes.
  • Contributor documentation describes the diagnostic mode and its limits.
  • Repository text contains no deployment history or private evidence.
  • Commits are signed off.

@vercel

vercel Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
loom-docs Ready Ready Preview Nov 21, 2026 1:28am UTC

Request Review

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@zlareb1
zlareb1 force-pushed the diag/pytest-native-signatures branch from 89e6059 to aeb1f66 Compare November 21, 2026 01:25

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant