Skip to content

fix: preserve memory when upgrading an existing database - #1787

Merged
santoshkumarradha merged 3 commits into
devfrom
fix-memory-upgrade-owner
Oct 7, 2026
Merged

santoshkumarradha merged 3 commits into
devfrom
fix-memory-upgrade-owner

Conversation

@santoshkumarradha

@santoshkumarradha santoshkumarradha commented Oct 7, 2026 •

Copy link
Copy Markdown
Member

Problem and fix

Upgrading an existing memory database to the contextual-memory release could disable memory at startup with initialize memories schema: no such column: owner, even when memory was enabled. The schema created the owner index before the migration added its column.

Create the index in the owner migration after the column exists, including fresh databases and databases missing only the index. On each open, safely assign owners to empty-owner rows left by older running builds, preserving nonempty owners and quarantining ambiguous project scopes.

Examples covered

  • Upgrade an existing database without losing notes, source references, status, counters, or scope.
  • Find previously saved notes through the old release's search index after upgrading.
  • Reopen repeatedly without rewriting correctly owned notes.
  • Recover notes an older build wrote into an upgraded database.
  • Open through the real chat startup path with memory available.

The built-in manual explains the error and recovery. Hosted workspaces need the affected engine restarted after updating, when ongoing work is safe to interrupt; closing a TUI window alone may leave that engine running.

Validation

  • Final Spark acceptance job 20261007-025237-001754-memory-upgrade-final-acceptance: make build and make pr-ready passed on 335ddd332a30496ae7c2f6f2e8d92d974994f11c, with no initial test failures. GitHub checks passed on the same head.
  • Live Spark job 20261007-025439-001756-memory-upgrade-final-hosted-live, exact deepseek/deepseek-v4.1-flash, no fallbacks: the default hosted engine opened a synthetic v0.7.1 memory database, recalled its saved shipment reference, saved a cutoff from ordinary conversation without an explicit remember request, and recalled it in a distinct new chat. The observed save was the model choosing the remember tool; this does not separately prove background extraction. The final answer treated the cutoff as a remembered statement, not an approved rule.
  • The live test confirms engine PID and an answering socket, then confirms its isolated engine and UI stopped. Debug and full-call recording switches are disabled because they bypass hosting.
  • Regression tests reproduced the original startup failure before the fix and test public store/chat startup, faithful released search-index data, repeated opens, owner isolation, and older-writer repair.
  • Parallel DeepSeek V4.1 Flash migration, test and final reviews completed. Final review's recovery-documentation findings were corrected; runtime code was unchanged afterward.

No production database was edited and no Blackmac engine was interrupted. Recovery requires the fixed build to open the database; an already-running engine without memory does not gain it merely because a binary was downloaded. Reconnecting a newer local build can replace an older busy engine, so update/reconnect when current work can be interrupted.

…ws on every open

The owner index was declared in memoriesSchema, which runs before
migrateMemoriesOwner adds the owner column. A database written before owners
existed already has the memories table, so CREATE TABLE IF NOT EXISTS left it
alone and the index was created against a column that was not there yet: every
existing store failed to open with `initialize memories schema: no such column:
owner`, and the launch carried on with no brain.

The index now lives beside the column it indexes, as memoriesOwnerIndexDDL in
the owner migration, and the migration runs its scope-derived backfill on every
open: the column is added on a legacy store, the index is ensured whatever shape
the table is in, and rows with an empty owner are owned again (or quarantined)
without touching a row that already has one. That last part is for the engines
that are still running an older build against a store this build already
upgraded: they never set the column, so their rows would otherwise be invisible
to every owner-filtered read forever.

The lexical index is deliberately untouched: a released store carries the
memories_fts view its own build maintained, and the regression seeds it with that
build's own refreshMemoryFTS statement rather than rebuilding anything. A store
whose view is absent keeps its notes and answers search with nothing, which is
the pre-existing behavior and is pinned by a test instead of changed.

Tests are built on a hand-written pre-owner database, not on today's schema:
internal/store covers Store.Open on the legacy shape (notes, provenance, status
and scope preserved; owner backfilled from scope; unprovable rows quarantined;
search matching and owner filtering still answer from the released index;
repeated opens identical; new stores indexed; already-owned stores without the
index repaired; empty rows a pre-owner writer inserted owned with every nonempty
owner left alone), and cmd/codeaf covers the real launch path through v3Memory.
The manual's "memory is off" page answers the sentence a person saw — the file is
intact, a local window is repaired by reopening, and a window held by an engine is
repaired by updating codeaf, stopping that engine with
`codeaf engine --stop --workspace <folder>`, and reconnecting — with a retrieval
probe pinning the exact error to that section.
… wording

The 'no such column: owner' section told a person that a local window keeps
its engine until the engine is stopped, and that a same-build engine is never
displaced. For a local engine that is false: a newer build replaces an older
engine on the connection, busy or not (engine.go clearStaleEngineHostAs ->
enginehost.Stop; TestASameProtocolOlderBuildIsReplacedBusyOrNot), while
installation alone changes nothing about an engine already running. Reword it
so the same machine repairs itself by updating and reconnecting, an engine on
another machine needs a targeted 'codeaf engine --stop --workspace <folder>'
(reached through ssh, not the local takeover), and an idle same-build engine
may still retire for a changed terminal environment. Drop the gratuitous
--stop-all instruction.

The upgrade preserves each note's words, provenance and status, but a project
row that cannot prove its project gains the legacy-project quarantine marker
and needs a proven owner; stop claiming its tags are untouched. Both claims
now match internal/store/memory_owner.go and the manual law.
@santoshkumarradha
santoshkumarradha marked this pull request as ready for review October 7, 2026 02:58
@santoshkumarradha
santoshkumarradha merged commit 0dbeccf into dev Oct 7, 2026
7 checks passed
@santoshkumarradha
santoshkumarradha deleted the fix-memory-upgrade-owner branch October 7, 2026 02:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant