Fix CPU allocation budgets and native worker aborts (5.0.1)

[mikessh]

Numerical runtimes and delegated sequence kernels could each claim a full CPU allocation. Native thread-creation failure also destroyed joinable threads and aborted Python. Version 5.0.1 validates one affinity/scheduler/cgroup budget, keeps persistent numerical pools serial, rejects excessive or nested kernel requests, and joins native workers before propagating exceptions.

Sample and corpus process pools are replaced with standard-library bounded threads. Frozen resources are shared, worker globals and spawn environment mutation are removed, and deferred readers can be closures. One-worker mode uses serial inner kernels without starting a pool. K-mer work with threads=1 runs on the calling thread.

Validation: 1,264 tests passed with 12 documented skips; focused pool/resource/corpus checks, exact serial/parallel outputs, native cleanup, clean-wheel installation and warning-free docs passed. Forced Linux thread-creation refusal aborts 5.0.0 and raises a catchable exception in the candidate. Fresh synthetic 48-CPU resource and scaling comparisons are being finalized before release promotion.

Initialized multi-thread Polars pools require a fresh interpreter before outer thread work. Existing BLAS idle threads cannot be destroyed by limiting compute budgets. The corpus reference sampler retains its seeded stream and remains GIL-limited. No live deployment diagnosis is claimed.

@copilot Please review the bounded compute budgets, shared thread workers and serial inner kernels, native worker exception/cleanup paths, and cgroup allocation handling. Check for oversubscription, races, deadlocks and behavior regressions.

🟡 Changes recommended

The GIL-bound corpus sampler loses process-level parallelism at the default build scale, and CLI help still advertises obsolete process workers.

2 open findings

• Retain process parallelism for GIL-bound pool generation · New
• Update --jobs help text to describe thread-based workers · New

What changed in this PR

Centralizes CPU budgeting, prevents native worker aborts, and moves Python workloads to bounded thread pools.

Changes:

• Adds allocation-aware CPU limits and serial inner-kernel enforcement.
• Joins native workers before propagating exceptions.
• Updates APIs, documentation, versioning, and regression tests.

File	Description
README.md	Documents thread-based CPU budgets.
docs/​signature.rst	Updates signature parallelism guidance.
docs/​usage.rst	Documents CPU/container behavior.
include/​vdjtools/​inext.hpp	Documents serial native default.
include/​vdjtools/​model.hpp	Updates E-step threading contract.
pyproject.toml	Bumps version and adds threadpoolctl.
python/​vdjtools/​__init__.py	Configures runtimes during import.
python/​vdjtools/​biomarker/​association.py	Applies validated kernel budgets.
python/​vdjtools/​biomarker/​cooccurrence.py	Applies validated kernel budgets.
python/​vdjtools/​biomarker/​fisher.py	Applies validated kernel budgets.
python/​vdjtools/​biomarker/​metaclonotype.py	Applies validated kernel budgets.
python/​vdjtools/​cores.py	Implements allocation detection and nesting controls.
python/​vdjtools/​dynamics/​groups.py	Validates delegated thread counts.
python/​vdjtools/​io/​batch.py	Bounds sample thread pools.
python/​vdjtools/​io/​convert.py	Resolves translation thread budgets.
python/​vdjtools/​model/​data.py	Bounds annotation and model-build concurrency.
python/​vdjtools/​overlap/​alice.py	Validates delegated thread counts.
python/​vdjtools/​overlap/​fuzzy.py	Validates fuzzy-search thread counts.
python/​vdjtools/​overlap/​similarity.py	Validates similarity-kernel budgets.
python/​vdjtools/​overlap/​tcrnet.py	Validates TCRnet thread counts.
python/​vdjtools/​preprocess/​correct.py	Budgets pairwise correction work.
python/​vdjtools/​signature/​cohort.py	Replaces process workers with threads.
python/​vdjtools/​signature/​corpus.py	Shares corpus resources across threads.
python/​vdjtools/​signature/​signature.py	Updates cohort worker contracts.
python/​vdjtools/​stats/​functional.py	Budgets similarity computation.
src/​_bindings.cpp	Documents native serial defaults.
src/​inext.cpp	Safely joins iNEXT workers.
src/​kmer.cpp	Adds serial mode and safe exception propagation.
src/​pgen.cpp	Safely joins and rethrows batch failures.
src/​thread_joiner.hpp	Adds native worker RAII joining.
tests/​cpp/​test_core.cpp	Tests joining during unwinding.
tests/​python/​conftest.py	Initializes runtime limits before collection.
tests/​python/​test_aa_scenarios.py	Makes thread tests allocation-aware.
tests/​python/​test_audit_contracts.py	Stabilizes build-budget expectations.
tests/​python/​test_cohort_stats.py	Bounds worker-count tests.
tests/​python/​test_cores.py	Tests nested serial budgets.
tests/​python/​test_generate_native.py	Bounds generation thread tests.
tests/​python/​test_infer_nt_batch.py	Bounds inference scaling tests.
tests/​python/​test_model_score.py	Bounds scoring thread tests.
tests/​python/​test_native_degenerate.py	Bounds degenerate batch tests.
tests/​python/​test_native_pgen_batch.py	Bounds native Pgen tests.
tests/​python/​test_pgen_nt_batch.py	Bounds nucleotide Pgen tests.
tests/​python/​test_signature_vectorized.py	Removes obsolete process state assertion.
tests/​python/​test_signature_workers.py	Tests shared threaded resources.
tests/​python/​test_thread_budget.py	Adds allocation and pool regressions.
tests/​python/​test_thread_scaling.py	Skips unsupported scaling environments.
tests/​python/​test_translate_junctions.py	Bounds translation thread tests.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Verified: the unchanged reference sampler is GIL-limited, so bounded threads do not accelerate pool generation. The owner explicitly requested removing the helper/process pool in favor of simpler shared threads. We retain that design and unchanged reference seeds/output. Commit 610c1f7 labels the old process timings as historical, documents that full pool builds can take about an hour or longer, and points to --fetch for published corpora. Restoring processes or changing the reference sampler is outside this requested fix; no multicore pool-generation claim remains.

Fixed in 610c1f7: corpus --jobs help now describes worker threads, allocated-core mode, no-pool serial mode, serial inner kernels and GIL-limited reference sampling. CliRunner help assertions, Ruff, diff checks and the warning-free Sphinx build passed.

@copilot Please review the bounded compute budgets, shared thread workers and serial inner kernels, native worker exception/cleanup paths, and cgroup allocation handling. Check for oversubscription, r...

Reviewed all four commits. Found and fixed a mapped-pool lifetime regression in 93fea12: corpus callbacks retained mappings during scratch deletion, preventing cleanup on Windows. Shared pool references now clear before deletion, including exception paths; added four serial/threaded success/failure regressions.

No additional high-confidence race, deadlock, oversubscription, native cleanup, or cgroup-mapping defects found. Budgets remain per invocation, not a global reservation across independently launched concurrent calls.

Validation: 128 targeted Python tests and the C++ suite passed; CodeQL reported zero alerts. Final read-only review found no significant issues (the automated review backend was unavailable). Windows cleanup is covered by lifetime assertions but was not executed on Windows.