chore(scale): W2 worker/queue/incremental analysis and recovery (GATE B)

[parthrohit22]

Task

Advance §23 W2 (Scale) foundations: make analysis incremental and operable — worker/service separation, a job queue, content cache with invalidation, and repository-class budgets with failure recovery. This is the GATE B precondition for W3 evolution.

Rationale

Roadmap §23 W2 mission: "Make analysis incremental, distributed and operable." §24 GATE B ("Reliability before history"): "Full and incremental outputs converge; entity identity evaluation passes" — a hard prerequisite for W3 graph evolution (#172, #173). Today analysis is a single in-process worker (roadmap §01 baseline confirms this). Without W2, drift/impact (the actual moat) becomes noise. Filing now to queue behind GATE A + foundation work; do not start until those pass.

References

• Roadmap §23 W2, §24 GATE B.
• Current: apps/backend/app/workers/analysis_worker.py (cancellable, single-worker).
• Blocks: feat(intelligence): snapshot-to-snapshot change comparison (repository evolution) #172 (snapshot diff), feat(intelligence): impact / blast-radius query over the snapshot graph #173 (impact query) — both correctly held until this lands.

Decomposition

This parent remains the GATE B roadmap issue. The implementation is split into these reviewable tasks:

• chore(scale): introduce the W2 queue and control-plane boundary #324 — queue and control-plane boundary;
• chore(scale): add worker failure recovery and bounded retry semantics #325 — worker failure recovery and bounded retries;
• chore(scale): add content-cache invalidation keyed by canonical graph hash #326 — content-cache invalidation keyed by canonical graph hash;
• chore(scale): prove full and incremental analysis convergence #327 — full/incremental convergence and budget evidence.

Do not start the children until the stated GATE A and foundation prerequisites pass.

Implementation Notes

• Separate control plane from workers (leases, queue, resource budgets).
• Content cache + invalidation keyed by canonical graph hash.
• Repository-class budgets; deterministic recovery on worker death.
• No new product surface; operability only.
• Keep the existing cancellable DB-backed lifecycle (analysis_worker CANCELLED_CODE) as the contract.

Acceptance Criteria

• Worker runs behind a queue/control plane, not in-process only.
• Incremental analysis converges with full analysis for representative repos.
• Content cache invalidates correctly on revision change.
• Failure recovery restores job state without orphaning snapshots.
• CI passes; no regressions to cancellable lifecycle.