feat(plugin): dual-backend fallback (remote primary + local fallback) by lightzt99 · Pull Request #1129 · oceanbase/powermem

lightzt99 · 2026-07-01T11:20:07Z

Summary

Add optional local fallback server so the Claude Code hook keeps working when the remote PowerMem backend is unreachable. Default read/write goes to remote; on net error / timeout / 5xx, the hook retries on the local fallback.
When POWERMEM_FALLBACK_BASE_URL is unset, behavior is byte-identical to single-backend mode (fast path) — no state file written, no probe.
Circuit-breaker state persisted to ~/.powermem/fallback-state.json with up/down TTLs (clamp 5..300s) to avoid probing primary on every hook event.
POWERMEM_FALLBACK_DISABLED=1 is a runtime kill switch that decays to primary-only without editing runtime.env.
init.sh learns dual mode: POWERMEM_INIT_FALLBACK_BASE_URL (non-interactive) or a TTY prompt. write_runtime_dual writes both URLs + keys with shell-safe quoting.
skills/init/SKILL.md gains a 4th Question 0 option "Remote + local fallback" with a dedicated onboarding round (fallback URL, port-conflict handling, MCP disabled note).
SETUP.md trimmed (~1076 → ~947 lines): dropped PRE-CHECK / STEP BY STEP / FINAL VALIDATION / SUMMARY sections that were redundant with init.sh or conflicted with SKILL.md; E006 now points to scripts/preload-model.sh.
Replay/sync deferred — needs a server-side metadata filter / list endpoint; will follow up separately.

Env vars

Var	Purpose	Default
`POWERMEM_FALLBACK_BASE_URL`	fallback URL; empty = single-backend	empty
`POWERMEM_FALLBACK_API_KEY`	fallback API key	empty
`POWERMEM_FALLBACK_DISABLED`	runtime kill switch	`0`
`POWERMEM_FALLBACK_DOWN_TTL_SECONDS`	"primary down" cache TTL (5..300)	`30`
`POWERMEM_FALLBACK_UP_TTL_SECONDS`	"primary up" cache TTL (5..300)	`30`
`POWERMEM_FALLBACK_TRIGGER_5XX`	treat 5xx as fallback trigger	`1`
`POWERMEM_FALLBACK_LOG_FILE`	fallback event log path	`$DATA_DIR/powermem-hook.log`
`POWERMEM_INIT_FALLBACK_BASE_URL`	init.sh non-interactive fallback URL	empty

Fallback triggers

Failure mode	Triggers fallback?
client timeout (`context.DeadlineExceeded`)	yes
dial refused / DNS / TLS / conn-reset (`net.Error`)	yes
HTTP 5xx with `TRIGGER_5XX=1` (default)	yes
HTTP 5xx with `TRIGGER_5XX=0`	no
HTTP 4xx	no
HTTP 2xx	no (success)

Known limitations (v1)

No replication. Memories written to the fallback during an outage are not automatically replayed to the primary after recovery; the two backends diverge until you manually reconcile.
Stale reads. Searches during fallback read from the local backend, which may lag behind the remote.
No conflict resolution. No automatic deduplication if both backends receive writes independently.
MCP transport not covered. Fallback applies to the hook's REST calls only, not to MCP server transport; dual mode disables MCP.

Test plan

go test ./... in apps/claude-code-plugin/cmd/powermem-hook — 14 new unit tests (helpers, state TTL, routing, 5xx trigger, kill switch, single-backend fast path)
python3 apps/claude-code-plugin/tests/test_runtime_dual.py — 5 cases including shell-metachar quoting for write_runtime_dual
Existing make test-claude-hook-docker regression — single-backend fast path preserved, unaffected
make test-claude-hook-dual — new Docker dual-backend regression (docker/Dockerfile.claude-hook-dual, tests/regression/test_claude_hook_dual_backend.py, 8 tests: both-up → primary, primary-down → fallback, cached-down skips probe, recovery marks up, 5xx trigger on/off, kill switch, single-backend no state file). Passes in 11s with --network none.
Manual: POWERMEM_INIT_BASE_URL=http://remote:8001 POWERMEM_INIT_FALLBACK_BASE_URL=http://localhost:8848 sh init.sh writes dual runtime.env; subsequent hook events route to primary, fall back on primary failure, recover after DOWN_TTL.

Non-goals (v1)

No background replay daemon (hook is short-lived; daemon is a follow-up)
No dual-write / multi-primary / load balancing
No MCP transport fallback
No exponential backoff within a single hook invocation (one primary + one fallback, then exit)

Files

apps/claude-code-plugin/cmd/powermem-hook/fallback.go (new) — routing, state, helpers
apps/claude-code-plugin/cmd/powermem-hook/fallback_test.go (new) — Go unit tests
apps/claude-code-plugin/cmd/powermem-hook/main.go — searchMemoriesForPrompt / postMemoryRaw routed via doRequestWithFallback
apps/claude-code-plugin/scripts/init.sh — dual mode branch, write_runtime_dual, reconfigure prompt
apps/claude-code-plugin/scripts/common.sh — write_runtime_dual helper
apps/claude-code-plugin/skills/init/SKILL.md — 4th Question 0 option + dual onboarding round
apps/claude-code-plugin/SETUP.md — trimmed; new "Remote primary + local fallback" section
apps/claude-code-plugin/tests/test_runtime_dual.py (new) — shell tests
tests/regression/test_claude_hook_dual_backend.py (new) — Docker regression
docker/Dockerfile.claude-hook-dual (new) — regression image
Makefile — test-claude-hook-dual target

Add optional local fallback server so the hook keeps working when the remote PowerMem backend is unreachable. When POWERMEM_FALLBACK_BASE_URL is unset, behavior is byte-identical to single-backend mode. - Go: doRequestWithFallback routes search/write to primary, retries on fallback for net errors / timeouts / 5xx (5xx opt-in via POWERMEM_FALLBACK_TRIGGER_5XX, default on). Circuit-breaker state persisted to ~/.powermem/fallback-state.json with up/down TTLs to avoid probing primary on every hook event. - Shell: write_runtime_dual in common.sh + dual-mode branch in init.sh (POWERMEM_INIT_FALLBACK_BASE_URL or interactive prompt). - POWERMEM_FALLBACK_DISABLED=1 is a runtime kill switch. - Replay/sync deferred: needs server-side metadata filter support.

…preload-model.sh Removed sections that duplicate script behavior or conflict with SKILL.md: - PRE-CHECK & PREREQUISITES (uv/Python detection lives in common.sh) - STEP BY STEP PROVEN PATH (Method A conflicts with SKILL.md's "do not run source/developer flow"; Method B uses outdated `claude mcp add -- powermem-mcp stdio` — init.sh now uses `--scope user --transport http`; Method C duplicates the E0xx error guide) - FINAL VALIDATION STEPS (status.sh already covers health; write/search round-trip belongs in status.sh, not setup docs) - SUMMARY (placeholder text, no information value) E006 now points to `scripts/preload-model.sh` instead of inline ModelScope/HuggingFace commands — the script already auto-detects region and bridges into the HF hub cache. Net: 1076 → 947 lines. Remaining sections are either canonical reference (masking rules, dev-mode, dual-backend env vars, E001-E014 troubleshooting, systemd unit) or the SKILL.md entry point.

Add "Remote + local fallback" as a fourth Question 0 option and a dual flow that: - Asks 4 questions in one round (Server URL, API Key, Connection, Fallback URL). Connection drops the MCP option in dual mode — fallback only applies to the hook's REST calls, not the MCP transport. - Reuses Question 1-3 (Storage / LLM / Embedding) verbatim to configure the fallback server, so the agent runs the same question flow as a standalone local server. - Defaults the fallback URL to http://localhost:8848 (same port as a standalone local server; port is the user's decision). - Detects port conflicts by probing health on the fallback URL: if a healthy server is already there, state the conflict plainly and let the user decide (reuse vs. pick a different URL). Never silently restart or kill the existing server. - Runs two init.sh invocations: (1) start the fallback server, (2) write the dual runtime.env with POWERMEM_INIT_FALLBACK_BASE_URL. - Notes v1 limitations (no replication, stale reads, no conflict resolution, MCP not covered) and the kill switch (POWERMEM_FALLBACK_DISABLED=1). - Documents the removal path: re-run init picking "Remote" (single backend) to overwrite runtime.env and clear the fallback URL.

Adds an isolated regression test that runs the hook binary against two in-process fake PowerMem HTTP servers (primary + fallback) and verifies the circuit-breaker routing end-to-end. No LLM, no network (Docker --network none). Coverage: - both up → request routes to primary, state file primary_down=false - primary down → request routes to fallback, state primary_down=true - cached down within TTL → skip primary probe, go straight to fallback - recovery → state seeded with stale last_probe_at, primary back up → probe hits primary, state marks primary_down=false - primary 5xx (TRIGGER_5XX=1 default) → fallback triggered, state down - primary 5xx (TRIGGER_5XX=0) → no trigger, error surfaces, no state - POWERMEM_FALLBACK_DISABLED=1 → kill switch, no fallback, no state - POWERMEM_FALLBACK_BASE_URL empty → single-backend fast path, no state The fake server subclasses RecordingHTTPServer so the handler's self.server resolves to the controllable instance (with label / fail_search / fail_health), and supports stop()/restart() on the same port for down→up transitions. Adds docker/Dockerfile.claude-hook-dual (same uv+go multi-stage pattern as the no-LLM regression image) and the make target test-claude-hook-dual.

wayyoungboy · 2026-07-01T17:25:11Z

I found one draft blocker in the dual fallback init flow. apps/claude-code-plugin/skills/init/SKILL.md starts the local fallback by passing POWERMEM_INIT_BASE_URL=<fallback_url> to apps/claude-code-plugin/scripts/init.sh, but the local init path does not derive the port from that URL; it uses POWERMEM_INIT_PORT if set, otherwise 8848 if free, otherwise 8849-8899.

If the user chooses http://localhost:8849 while 8848 is free, init starts the server on 8848 but the next step writes POWERMEM_FALLBACK_BASE_URL=http://localhost:8849, so fallback points at a port where nothing was started. Please either pass/derive POWERMEM_INIT_PORT from the selected fallback URL, or change the skill/script flow to only write a fallback URL that was actually started and verified.

init.sh: POWERMEM_INIT_RECONFIGURE=1 stops the healthy managed server, removes .env, recreates it from current POWERMEM_INIT_* env, and starts a fresh server. Without the flag, init prints a hint and exits 0. SKILL.md: when status.sh reports the server healthy (local or remote), ask via AskUserQuestion whether to keep current setup or reconfigure. Local managed mode runs init.sh with RECONFIGURE=1; remote mode re-asks the Server URL / API key / Connection / Fallback questions and re-runs init.sh (remote server is not managed by the plugin, so no RECONFIGURE flag needed). Also rebuilds the 5 platform hook binaries to include the dual-backend code.

wayyoungboy · 2026-07-13T17:44:43Z

Thanks for the dual-backend fallback work. I do not think this is ready to merge yet because the fallback init flow can still write a URL that does not match the server that was actually started.

If the user selects a fallback URL such as http://localhost:8849 while port 8848 is free, the init script may start the local server on 8848 but still persist POWERMEM_FALLBACK_BASE_URL=http://localhost:8849. Please either derive/pass POWERMEM_INIT_PORT from the selected fallback URL, or only write a fallback URL after that exact endpoint has been started and verified.

lightzt99 added 4 commits July 1, 2026 17:30

lightzt99 marked this pull request as draft July 1, 2026 11:20

wayyoungboy marked this pull request as ready for review July 6, 2026 09:24

Navigation Menu

Uh oh!

feat(plugin): dual-backend fallback (remote primary + local fallback) - #1129

feat(plugin): dual-backend fallback (remote primary + local fallback)#1129
lightzt99 wants to merge 5 commits into
oceanbase:mainfrom
lightzt99:feat/dual-backend-fallback

lightzt99 commented Jul 1, 2026 •

edited

Loading

Uh oh!

wayyoungboy commented Jul 1, 2026

Uh oh!

wayyoungboy commented Jul 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

lightzt99 commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Env vars

Fallback triggers

Known limitations (v1)

Test plan

Non-goals (v1)

Files

Uh oh!

wayyoungboy commented Jul 1, 2026

Uh oh!

wayyoungboy commented Jul 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lightzt99 commented Jul 1, 2026 •

edited

Loading