sessions: re-import after store reset holds ~1.7 cores for 4+ hours; profile_sessions writes take 125–160 ms; restart repeats the import #4756

Summary

After I reset 7 project stores (schema v39 → v40), the daemon re-imports session history for more than 4 hours. It uses about 1.5–1.8 CPU cores the whole time. Each write to the profile session store takes 125–160 ms. A daemon restart does not help: the import starts again from zero, and the CPU goes back to about 1.75 cores within 30 seconds.

This is probably related to #4481 (still open, same machine). The trigger here is different: a large one-time re-import after a store reset. Also related: #729 (catch-up restarts from zero) and #4374 (per-rollout re-offer cost).

Environment

• tracedecay 1.0.0-beta.96+e7d26a204b136d84d8373237561c361fe50b9ec3.dirty (local build; the commit is identical to the v1.0.0-beta.96 tag)
• macOS 27.2 (26B123), Apple M4 (10 CPUs), 32 GB RAM
• Daemon runs as a launchd user service
• 12 registered projects
• Transcripts on disk (about 5 GB in total):

• Profile user-sessions.db: 1.3 GB

Steps to reproduce

1. Have project stores at schema v39. The beta.96 daemon refuses them with database schema v39 is not the v40 shape this binary creates.
2. For each of the 7 projects: run tracedecay projects forget <id> --yes, then run tracedecay init in the project.
3. Wait for the code index of each project to reach current. This took about 4 minutes.
4. Watch the daemon CPU for the next hours.
5. Run tracedecay daemon restart, and watch the CPU again.

Expected

The re-import finishes in a reasonable time and the daemon goes idle. After a restart, the import continues from where it stopped.

Actual

1. CPU stays high for more than 4 hours

The log shows only idle scheduler events (scheduler_interval_not_elapsed, no_new_session_activity). No errors.

All 7 projects show this status the whole time:

session_projection: stale (historical_retry:history_admission_saturated)
worker.backlog: 1

2. Writes to the profile session store are very slow

I compared two tracedecay_runtime shard snapshots taken 60 s apart, 4 hours after the restart:

The two slow shards together are about 1.8 cores of transaction time. This matches the process CPU.

The average time per write on profile_sessions went up over the same daemon run: about 59 ms (85,670 writes in 5,044 s), then about 125 ms. So the cost per write probably grows with the store size. This is not confirmed.

3. A restart repeats the import

After tracedecay daemon restart (CPU in cores, from ps CPU time deltas):

03:45:25 cores=2.16 rss_mb=1424
03:45:55 cores=2.83 rss_mb=4469
03:46:25 cores=1.83 rss_mb=4471
03:46:55 cores=1.76 rss_mb=4479
03:47:25 cores=1.73 rss_mb=4480
03:47:55 cores=1.76 rss_mb=4482

Shard snapshots 60 s apart, about 4 minutes after the restart:

Before the restart, that large project's session shard had 107,137 completed writes and wrote nothing in a 60 s window. After the restart it writes about 1,100 rows per minute again. The counters reset with the process, so I cannot prove that the same rows are written again. One live agent session ran in that project, but it does not explain 1,100 writes per minute. So the restart probably repeats the import for that project instead of continuing it.

RSS also goes from 1.4 GB to 4.5 GB in the first 30 s after the restart.

4. Other observations

• No observation_retrieval_anchor_alias_collision refusals since the restart. The old log has 31,997 of them from before this beta.96 run, so the sessions: admission refusal storm (observation_retrieval_anchor_alias_collision, ~68k WARNs, ~640 sessions) pins daemon at ~180% CPU #4564 fix seems to work.
• Since the restart there is only one slow-poll warning, at crates/tracedecay-session-runtime/src/session_temporal_refresh_scheduler/registry.rs:401:40.
• sample (8 s): most threads wait in __psynch_cvwait. The active work is spread over about 10 worker threads at about 10% each. The binary has no symbols, so I cannot name the functions.

Questions

1. Is a 125–160 ms write on profile_sessions expected for a 1.3 GB user-sessions.db?
2. Should a restart continue the historical import from a durable frontier, instead of starting again?
3. Can the user limit or pause the historical import? I found no configuration key for it.

I have not tested beta.97 yet. Its notes include #4706 (Codex dispatch and per-rollout phases), which can be related.