Skip to content

fix(storage): repair call log columns in versioned user databases - #283

Merged
loveRhythm1990 merged 2 commits into
matrixorigin:mainfrom
loveRhythm1990:fix/call-log-schema-upgrade
Oct 9, 2026
Merged

loveRhythm1990 merged 2 commits into
matrixorigin:mainfrom
loveRhythm1990:fix/call-log-schema-upgrade

Conversation

@loveRhythm1990

@loveRhythm1990 loveRhythm1990 commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

What type of PR is this?

  • fix (bug fix)

Which issue(s) this PR fixes

Follow-up to #258; no linked issue.

What this PR does / why we need it

Upgrading a user database created before #258 can leave mem_api_call_log without tool_success and tool_error_kind. Those columns were added while the user schema version remained 2, so migrate_user() returns early for existing version-2 databases. The new call-log writer then fails to insert monitoring entries, and call-statistics queries fail on the missing columns.

Check and repair the two nullable columns before the current-version fast path returns, and reuse the repair in the older-schema migration path. Existing columns are skipped. The ALTER statements use the existing bounded MatrixOne transaction retry helper; duplicate-column errors are harmless, and a secondary-index DDL race is accepted only if the requested column now exists.

For an otherwise usable version-2 database, failure of this optional monitoring repair is logged and does not block memory reads or writes. The background call-log writer retries schema repair on subsequent batches, including through already-cached user stores. A per-store success hint avoids repeated metadata queries after repair; failures remain retryable. As with existing failed call-log inserts, batches whose schema repair fails are dropped with a warning; they are not retained in an unbounded queue. Call statistics remain degraded until the log table can be repaired.

Keep the version marker unchanged for already-current databases. This targeted repair avoids replaying unrelated historical memory/branch migrations and remains compatible with old API instances during a rolling update: old writers omit these nullable columns, and historical rows retain NULL outcomes. User migration remains lazy, triggered when the database is opened.

Validation

  • Reproduced the original missing-column defect on the unmodified base.
  • Five SQL integration regressions passed on an isolated MatrixOne 4.2.1-d2393868a instance: version 1 and version 2, partial repair, concurrent instances, and a logging failure followed by recovery on the same cached store. They verify preserved history and memories, old/new writer compatibility, usable statistics, and idempotent reentry. Successful fixtures remove their owned shared/user databases.
  • Existing multi-database routing/migration regressions: 3 passed.
  • API end-to-end regression verifies memory writes and reads during a logging outage, followed by background schema repair, tool-outcome persistence, and working call statistics without cache invalidation.
  • Existing API end-to-end tool-outcome persistence regression also verified.
  • Storage/API library tests: 157 passed, 1 existing explicit DB test ignored. Strict Clippy for both libraries plus the new storage integration target passed with -D warnings. Strict Clippy of the entire existing API e2e target additionally exposes pre-existing type_complexity and to_string_in_format_args warnings outside this change.
  • Changed-line formatting and git diff --check passed.

No production schema changes or deployment were performed.

@loveRhythm1990

Copy link
Copy Markdown
Collaborator Author

Review(HEAD faf7a46)

结论:问题属实,修复方向正确,没有发现语法或逻辑错误,可以合。 下面有两条建议值得在合并前考虑,其余为小问题。

1. 问题是否属实 —— 属实

2. 修复方法是否正确 —— 正确

  • 在快速返回之前补这两列,和 ensure_author_id_column / ensure_subject_id_column 的处理方式一致。不改版本号,也不重放历史迁移,对滚动升级更安全。两列都是 nullable,旧实例写入时不带这两列也没问题。
  • 先用 information_schema 检查列是否存在再 ALTER。重复列错误(1060)直接接受;MO 二级索引的 1146 竞态,只有在复查确认列已存在时才接受。判断条件比原来更严,是对的。
  • router.rs 在 migrate_user 失败时不会写 user_schema_cache,所以修复失败不会被当成"已就绪"缓存下来,PR 描述的说法成立。
  • 快速路径每个实例、每个用户只走一次(受 user_schema_cache 控制),新增的两次 information_schema 查询开销可以忽略。
  • 慢路径改成调用同一个函数,原逻辑等价保留,并且多了"列已存在就跳过"和竞态容忍。

本地验证结果:cargo test -p memoria-storage --lib 51 个通过、1 个忽略;cargo clippy -p memoria-storage --lib --test call_log_schema_upgrade -- -D warnings 通过;本 PR 改动的行 cargo fmt --check 无差异(crate 里其他文件有一些既有的 fmt 差异,与本 PR 无关)。新增的 5 个 SQL 集成测试需要 MatrixOne,本地没有跑。

建议

(a) 失败后果比原问题本身更重(可讨论,建议处理)。
call log 是异步批量写的(memoria-api/src/auth.rs 的 flusher),写失败只会打 warn!。修复前缺列的影响只是监控数据丢失、统计查询失败。修复后,快速路径里 ALTER 一旦失败,migrate_user 返回 Err,这个用户的所有记忆接口都无法使用,直到某次重试成功为止。暂时性错误下次请求会自愈,但如果是持续性失败,就会把一个监控问题升级成用户级不可用。持续性失败的例子包括:DDL 权限不足、长时间锁冲突、表缺失(failed_log_repair_does_not_mark_user_schema_ready 正好把"表缺失 ⇒ 用户不可用"固化成了预期行为)。

建议快速路径对这一步采用"记录 error 后继续":

  • 例如在快速路径里 if let Err(e) = ... { tracing::error!(...) },而不是用 ? 直接返回;
  • 或者至少在表不存在时跳过(ensure_subject_id_column 对 mem_memories 就是这么做的)。
  • 如果担心失败后不再重试,可以在失败时不写 user_schema_cache,但仍然返回可用的 store。

subject_id 必须 fail-fast,是因为缺了它读写会出错;这两列只影响监控,用不着同样的严格程度。

(b) ALTER 没有重试 MO 的 20631("txn need retry in rc mode, def changed")。
文件里已经有 exec_ddl_with_retry,ensure_subject_id_column 等也在用。这一步现在挡在用户可用性的路径上,建议把 sqlx::query(...).execute(pool) 换成 exec_ddl_with_retry(pool, &sql),以减少和 branch merge 等 DDL 并发时的偶发失败。如果采纳了 (a),这条就只是锦上添花。

小问题

  • 测试的 fixture 会为每个用例创建 call_log_upgrade_<uuid> 共享库和用户库,但从不删除,CI 的 MO 实例会不断累积测试库。如果其他 SQL 测试有清理约定,建议跟着做。
  • ensure_call_log_tool_outcome_columns 的注释写的是 "added after schema version 2",更准确的说法是"在版本 2 之后加入但没有提升版本号",和 ensure_subject_id_column 的注释措辞保持一致即可。不影响正确性。

@loveRhythm1990
loveRhythm1990 merged commit 3a5c506 into matrixorigin:main Oct 9, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant