Skip to content

fix(v0.5.0): core owns the knowledge importer; resume after quit; reset log line (C3) - #275

Merged
SaulBuilds merged 1 commit into
release/0.5.0-hermes-upskillfrom
fix/v0.5.0-importer-lifecycle
Oct 7, 2026
Merged

SaulBuilds merged 1 commit into
release/0.5.0-hermes-upskillfrom
fix/v0.5.0-importer-lifecycle

Conversation

@SaulBuilds

@SaulBuilds SaulBuilds commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Fixes the v0.5.0 release blocker from the DGX C3 Linux run (#266, findings 1 and 2). Base: release/0.5.0-hermes-upskill (head 69f8c1a, includes #274).

1. The knowledge importer outlived quit (C3 check 4 FAIL)

knowledge_import::run_importer spawned mem-mcp import-corpus as a plain child that shutdown_all_sidecars did not stop. Quitting left it running and holding the memory store's RocksDB LOCK, reparented to PID 1.

  • ImportRegistry now owns the importer child while it runs. adopt and shutdown share a gate, so every importer is either adopted before quit, and stopped by it, or refused after it.
  • shutdown_all_sidecars, which runs on ExitRequested, Exit and "Delete my local data", now calls knowledge_import::shutdown() first. That call closes the registry, sends SIGTERM to each running importer, sends SIGKILL after the supervisor's stop grace (DEFAULT_STOP_GRACE, 5 s) and reaps the process. A second call does nothing.
  • Once the registry is closed, no new import starts and the memory daemon is not restarted after an interrupted import. The restart runs under the gate, so it cannot race the quit and leave an orphaned daemon.
  • MemoryManager::start refuses with an honest error while an importer holds the store. The daemon could not take the lock and would crash-loop. I chose to refuse rather than kill the import, because the import restarts the daemon itself when it finishes. The UI already waits for the import before it starts the daemon.
  • kit: DEFAULT_STOP_GRACE is now public, and there is a new terminate_unsupervised() for one-shot children. It reuses the supervisor's SIGTERM, then grace, then SIGKILL path, and skips an already-reaped child.

2. Resume (behavior chosen)

mem-mcp import-corpus at memories 0e9d488 (the pinned binary, unchanged here) already resumes per tenant. It writes each tenant's bundle hash into store meta only after the whole tenant has landed, and on the next run it skips tenants whose hash is recorded. Merging nodes it already wrote is a CRDT no-op. It does not skip nodes inside an unfinished tenant, so the tenant that was in progress starts over.

What core adds:

  • An interruption is reported as failed with a clear message: the import stopped because the app quit and continues on the next launch. No completion marker is written, so the next launch runs the import again and the importer skips the finished tenants.
  • memory/knowledge-corpus.progress.json records the finished tenants, the tenant in progress with done/total, and whether the run was interrupted or failed. It is removed when the marker is written. A resumed run logs which tenants a previous run already finished.

Live proof (#[ignore] test live_interrupted_real_import_resumes_and_leaves_the_store_consistent), run on the release Mac with the pinned mem-mcp (31598435…), the release corpus (815eda93…, 32,754 nodes) and the full BGE:

  • Run 1: stopped after 3.6 s, with citrate-docs and methodology finished and refs at 512 of 12,730.
  • Run 2: the store opened, so the lock had been released. It skipped citrate-docs and methodology, imported refs and skills (30,411 nodes, 29,691 edges, 0 embedded, 30,977 vectors reused) and finished in 271 s.
  • Run 3: with the marker removed, it added 0 nodes and 0 edges, so the store was consistent.

Left for 0.5.1: node-level resume inside a tenant (the importer would need to skip node ids already in the store, a citrate-memories change). The worst case now is redoing one tenant. skills, the largest, is 18,247 nodes. This is stated in docs/releases/v0.5.0.md known issues.

3. Reset log line (C3 finding 2)

The text after "key material kept:" listed the removed files, and a missing \ continuation left a run of spaces. The line is now built by node_genesis::reset_log_message:
[node] chain genesis changed (P -> G): removed N chain database file(s), B bytes, from DIR; removed: …; kept: …
The kept list is read from the data dir after the reset (key material, node.toml, chain-genesis, …).

Tests

New tests (each one cleans up every process it starts; a Reaper kills any leftover pid even when an assertion fails):

  • app_quit_stops_a_running_importer_and_reaps_it: the pid is dead when shutdown returns, the run is reported as interrupted, no marker is written, and the progress record lists the finished tenants.
  • an_importer_that_ignores_sigterm_is_killed_after_the_grace
  • an_interrupted_import_continues_on_the_next_launch_without_redoing_finished_tenants
  • no_importer_starts_once_app_quit_began
  • app_quit_during_an_import_does_not_restart_the_memory_daemon
  • the_memory_daemon_refuses_to_start_while_an_importer_holds_the_store
  • the_quit_hook_stops_the_importer_before_any_sidecar (tripwire)
  • node_genesis::the_reset_log_line_lists_removed_and_kept_files_correctly
  • The live #[ignore] test above.

Mutation check: with the terminate call in ImportRegistry::shutdown disabled, app_quit_stops… and …ignores_sigterm… fail.

Test-only change: memory::tests::restart_is_safe_with_a_leftover_socket_file gave its fixture 500 ms. It also failed under parallel load on the base branch, so the bound is now 5 s.

Gates (release Mac, 1.98.1)

  • cargo fmt --all -- --check: clean
  • cargo +1.98.1 clippy --workspace --all-targets --locked -- -D warnings: clean
  • cargo +1.98.1 test --workspace --locked: 2,374 passed, 0 failed, 20 ignored (21 suites). rc.2 was 2,366 passed and 19 ignored.
  • npm run typecheck: clean
  • vitest: 2,459 passed, 35 skipped (268 files). The total is the same as rc.2 (2,494). The 11 extra skips are tests that need the federation source checkouts (skills.lock sources, the qa/literacy pinned sources), which they look for relative to the repo, and a nested worktree does not have them. No test failed. There is no TypeScript change.

🤖 Generated with Claude Code

https://claude.ai/code/session_018wWZZ8GRVU9USh3kKQHsFe

…et log line (C3)

DGX C3 Linux run (#266, findings 1 and 2):

1. `mem-mcp import-corpus` outlived app quit holding the memory store's
   RocksDB LOCK. ImportRegistry now owns the importer child: the quit hook
   (shutdown_all_sidecars, every exit path) calls knowledge_import::shutdown()
   first, which closes the registry, SIGTERMs each running importer, SIGKILLs
   after the supervisor stop grace (5 s) and reaps it. Once closed no importer
   starts and the memory daemon is not restarted after an interrupted import.
   MemoryManager::start refuses while an importer holds the store.
   kit: DEFAULT_STOP_GRACE is public; terminate_unsupervised() for one-shot
   children.

2. Resume: mem-mcp 0e9d488 records each tenant's bundle hash only after the
   tenant landed and skips recorded tenants, so an interrupted import already
   continues at tenant granularity (the in-flight tenant starts over). Core
   now reports an interruption honestly (no marker, re-run next launch) and
   keeps memory/knowledge-corpus.progress.json (finished tenants, in-flight
   tenant, interrupted/failed), removed when the marker is written. Live proof
   with the pinned mem-mcp and the release corpus: stopped after 3.6 s mid
   refs; the re-run skipped citrate-docs and methodology and finished in
   271 s; a third run added 0 nodes.

3. node.rs reset log line: the list after "key material kept:" was the
   removed files and a missing line continuation left a run of spaces. Now
   "removed: ...; kept: ..." with the kept list read from the data dir;
   node_genesis::reset_log_message is tested.

Test-only: memory::tests::restart_is_safe_with_a_leftover_socket_file waited
500 ms for its fixture and failed under parallel load on the base branch too;
the bound is now 5 s.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018wWZZ8GRVU9USh3kKQHsFe
@SaulBuilds
SaulBuilds requested a review from a team as a code owner October 7, 2026 02:16
@SaulBuilds
SaulBuilds merged commit 02cc328 into release/0.5.0-hermes-upskill Oct 7, 2026
5 checks passed
@SaulBuilds
SaulBuilds deleted the fix/v0.5.0-importer-lifecycle branch October 7, 2026 02:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant