fix(memory): honour project:<id> isolation + make the degrade ladder real (#1300, #1301)

[thinmintdev]

Closes #1300. Closes #1301.

#1300 — cross-project memory bleed on every default install

In unified_bank mode (the shipped default) _write_namespace collapsed everything except agents onto the single shared bank. Unlike private:, which survives the collapse as a visibility:private + agent:<id> tag pair enforced on read, project: got no compensating marker at all. A project:foo write was literally an ordinary shared write; a project:bar recall returned it.

Chose to honour the isolation, not reject it. project:<id> is in the closed spec §3 namespace table accepted by both REST and MCP front doors, unified_bank=True is the default, and memory_delete already documents sweeping by it — refusing it in unified mode would delete a documented, front-door-reachable feature on every default install.

The namespace now survives as a server-owned project:<id> tag, enforced by the same mechanism visibility:private already uses (PR #1244) — one pattern in the file, not two.

• add stamps the resolved scope and strips caller-supplied project: tags (otherwise a shared write can forge its way into another projects recalls)
• recall/search/list_items filter pre-rerank and pre-budget, so out-of-scope docs never consume the callers tokens
• delete gates on the same predicate
• agents stays exempt; unified_bank=False untouched

Migration: docs written as project:foo before this were actually shared writes — that was the bug — and nothing can distinguish them, so there is nothing to back-fill.

#1301 — the ladder never engaged and the flag lied

_build_hindsight_client only called HindsightRestClient.from_env(), which constructs an httpx client and does zero I/O. A down daemon never raised, so the except in provider_from_config was dead code. Proven on base:

boot with DEAD daemon -> provider: HindsightProvider
degraded flag reports: False
runtime recall failed: connection refused
degraded after engine death: False

Post-fix: PgVectorProvider / True / True.

Fixed both halves — a boot-only probe would start lying again the moment the daemon died:

• Boot: probe_hindsight() does a bounded synchronous GET /health. "Reachable" = HTTP response below 500, so a 401/404 proves something is listening and must not trigger a data-losing degrade. Escape hatches: HAL0_HINDSIGHT_PROBE_TIMEOUT_S, HAL0_HINDSIGHT_SKIP_PROBE
• Runtime: degraded is a live property driven by observed transport failures, clearing on recovery

Verification

• 26 new tests; 9 fail on base, including the cross-project bleed and the tag-forgery case
• tests/memory → 144 passed; combined with api/dispatcher → 419 vs base 393 (exactly +26)
• tests/cli → 629, identical to base
• ruff / format / sunset / import smoke clean

The ladder test uses a real closed loopback port and a real throwaway HTTP daemon rather than patching httpx — "no I/O happened" was the bug, so a test that stubs the I/O could not have caught it.

🤖 Generated with Claude Code

[thinmintdev]

⚠️ Converted to draft — collision detected.

610150d7 ("fix(tests): hermes uninstall tests hit real system bus; RC 1.0 carryover", #1357) landed on main and already closes #1302, #1301, #1300, #1224 via an independent implementation.

This PR is being reconciled against it. It will either be closed as superseded or reduced to the genuinely unique delta (e.g. the security/exposure.py ADMIN route pinning and #1024, which 610150d7 does not appear to cover). Not merging duplicate implementations.

[thinmintdev]

Reduced after reconciliation with 610150d7

Rebased onto main, cut 708 → 325 lines.

Dropped as duplicate — and 610150d7s versions are better:

• memory: project:<id> isolation is a silent no-op in unified_bank mode #1300 project scoping. Mains _projects_of returns a set with OR semantics (a doc can be filed under several projects) and only stamps the tag if self._unified_bank, leaving pre-unified multi-bank deployments untouched. This branch stamped unconditionally. Its extra strictness (stripping caller-supplied project: tags) is a reasonable disagreement, not a defect — main treats multi-project filing as deliberate. Dropped.
• memory: hindsight→pgvector degrade ladder is inert; 'degraded' flag is misleading #1301 boot probe. Mains probe_health sends the Authorization header and takes an injectable transport; this branchs probe_hindsight did neither. Dropped.

Kept: the flag is still boot-only on main

grep degraded over mains hindsight_provider.py returns nothing. HindsightProvider has no degraded attribute, so _memory_degradeds getattr(provider, "degraded", False) is False for the entire life of the process.

A boot probe answers one question once and the answer goes stale immediately. A daemon that dies after boot leaves HindsightProvider in place, and /api/status.memory_degraded plus hal0 memory status go straight back to reporting healthy while recalls return empty and retains raise — the same false-healthy report #1301 was filed about, displaced by one boot. A boot probe also structurally cannot clear itself on recovery.

Surviving delta:

• _degraded + live degraded property + _engine_answered + _mark_engine (edge-triggered logging) + a _call wrapper, with all five client round-trips (retain, recall, list_memories ×2, delete_document) routed through it
• 4xx does not degrade — the delete sweeps routine per-bank 404s must not flap the flag; 5xx does, matching probe_health
• Corrected health.py and hal0 memory status wording, which asserted "in-memory fallback" for what may be a failing durable engine

Anti-vacuity guard kept: mains test_hindsight_boot_probe.py monkeypatches probe_health itself, so it cannot distinguish "the probe ran and failed" from "the probe never reached the network" — the exact shape of the original bug. The added test points the factory at a real closed loopback port.

Remaining gap, flagged not fixed

The boot decision is still permanent in the other direction: a box that starts hal0-api before hindsight-api degrades to volatile PgVector at boot and never re-promotes.

Verification: ruff check ✅ · ruff format ✅ · sunset ✅ · tests/memory tests/api tests/cli tests/mcp → 2466 passed, 1 skipped, 2 deselected.