fix(tokenizer): keep the tiktoken daemon thread out of the dynamic linker

[zhanghanduo]

What

runtime/loop/tokenizer.py spawned a daemon thread whose first statement was import tiktoken. This moves the import to the calling thread and leaves only tiktoken.get_encoding() on the thread, plus an atexit join for an in-flight load.

Why

tiktoken._tiktoken is a Rust extension, so that import dlopens a shared object. CPython kills daemon threads mid-flight during finalization (pthread_exit at the next GIL acquisition), and being killed inside the dynamic linker is not survivable:

• losing glibc's _dl_load_lock → later SIGSEGV
• unwinding through a Rust / extern "C" frame → abort() → SIGABRT

Both were observed in ApodexHarness as a signal death after a fully correct protocol stream (run_finished status=success + final both on stdout, stderr empty):

test	exit
tests/sdk_cli/test_protocol_compliance.py::test_stateless_across_invocations	-11
tests/sdk_cli/test_serve_command.py::test_serve_subprocess_e2e	-6

An all-thread faulthandler dump registered at atexit (i.e. the last moment Python code runs, before daemon threads are killed) showed the thread parked inside that import on every short run:

File "<frozen importlib._bootstrap_external>", line 147 in _path_stat
...
File ".../tiktoken/__init__.py", line 3 in <module>
...
File ".../agent_core/runtime/loop/tokenizer.py", line 51 in _load
File "/usr/lib/python3.12/threading.py", line 1010 in run

Not a rare race: the import takes 24-29 ms and the first get_encoding_nonblocking() call lands late in a run, so a short process reliably exits inside the window. It is usually non-fatal, which is why it read as a flake. The window is wider wherever TIKTOKEN_CACHE_DIR is unset (CI does not set it, the product Dockerfiles do), because the load then takes the no-timeout network path.

Why the import can move but get_encoding cannot

The module's stated defense is against a minutes-long, no-timeout HTTP fetch freezing the event loop (the 2026-06-05 182 s hang, the 2026-06-08 swarm-gv wedge). That fetch is in get_encoding(), which stays on the daemon thread. The import is local: 24-29 ms, no network. So the intent is preserved.

The atexit join follows the existing precedent in providers/nonblocking_stream.py, which already joins its daemon thread with a 1 s timeout — this module was the one spot that omitted it.

Behaviour change for consumers

None to the contract: get_encoding_nonblocking() keeps its signature, still returns None while loading, callers still fall back to a heuristic. One timing difference: the first call for an encoding name now spends 24-29 ms importing on the calling thread where before it returned instantly. In exchange the encoder lands sooner, shortening the heuristic-fallback window tokens.py documents as "the opening turns of every process".

Tests

tests/test_tokenizer_nonblocking.py (new, 5 cases). The regression guard installs a sys.meta_path finder that records which thread loaded tiktoken and which called get_encoding, and asserts import-on-caller / encode-on-daemon. Also covered: the atexit hook actually drains and the thread deregisters itself; a wedged get_encoding never blocks the caller; missing tiktoken is terminal and spawns no thread; concurrent callers spawn exactly one thread.

Local: ruff clean, pyright 0 errors, check_unconsumed_fields clean, 1479 passed, 1 skipped, version bumped 0.8.0 → 0.8.1 with a CHANGELOG entry.

Honest limitation

The crash itself was not reproduced on the investigation host (420 isolation runs across four harnesses, 0 non-zero exits — Linux 5.10 / glibc / CPython 3.12.3). The mechanism is the only hypothesis that explains both -11 and -6, and the mid-import park is 100% reproducible, but this is a fix for a diagnosed hazard rather than a fix verified by a now-passing reproducer.

Full investigation record: temp/2026-09-05_serve-post-final-signal-death.md in ApodexHarness.