Skip to content

fix(conversations): a conversation never mixes forum topics - #3107

Merged
leemour merged 2 commits into
mainfrom
fix/conversations-forum-topics
Nov 18, 2026
Merged

leemour merged 2 commits into
mainfrom
fix/conversations-forum-topics

Conversation

@leemour

@leemour leemour commented Nov 17, 2026

Copy link
Copy Markdown
Member

Task

Conversations built from a Telegram forum group (a group with topics) mixed the topics: in one measured forum, one
conversation held most of the group's messages over four years, and a fifth of its chunks held lines from several
topics, so search by meaning read unrelated topics as one text.

Why

In a Telegram forum every plain message of a topic replies to the topic's first message, an empty service message
with no topic of its own. The builder took that as a real reply and chained each topic into one conversation; the
first messages of all topics, written by one person minutes apart, were then joined by the same-sender rule, so every
topic ended up in one conversation.

What

  • A message's parent is a message of its own topic or thread, or the thread's first message when that has text (a
    WhatsApp community post, which its comments carry as their thread and reply to). An empty first message, as a
    Telegram topic's, is no parent. A messenger reply into another topic and an agent's answer naming one are no link;
    the rules already stayed inside a topic.
  • RULES_VERSION 5 → 6: every chat built before reads as outdated and is rebuilt. Chats without topics get the same
    conversations and chunk hashes, so their vectors are kept.
  • Agent batches: a message whose only messenger reply the build does not link is now asked about.
  • The thread graph (graph.reply in message metadata) is untouched: tg already leaves the topic root out of it.
    messages context --thread already stopped at a topic boundary.

Testing

  • Unit tests on invented data: topics kept apart, a comment kept on its post, cross-topic replies and agent answers
    ignored, agent batches.
  • On a copy of a test store (one forum, about 112k messages, 26 topics), measured before/after: chunks mixing topics
    5,700 of 25,401 → 0 of 42,639; conversations mixing topics 14 → 0; largest conversation 42,601 → 207 messages.
    None of the 30 stored topic first messages has text. Search checks on five questions did not change rank.
  • Local: pnpm ci:quick, Biome, pnpm docs:check, vitest over src/conversations, src/store, src/services.

🤖 Generated with Claude Code

leemour and others added 2 commits November 18, 2026 00:50
A message's parent is now always a message in its own topic or thread. A
messenger reply into another topic, an agent's answer naming one, and the
rules (as before) never link across.

In a Telegram forum every message of a topic replies to the topic's first
message, which carries no topic itself. Taken as a reply, it chained a whole
topic into one conversation, and the first messages, written by one person
minutes apart, were joined by the same-sender rule, so every topic of the
forum became one conversation whose chunks mixed topics.

RULES_VERSION is 6, so chats built before read as outdated and are rebuilt;
a chunk whose text did not change keeps its vector (embeddings are keyed by
the chunk's hash). The agent batches ask about a message whose only reply
points into another topic, as the build no longer counts it as replied.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e with text

A WhatsApp community comment carries the post as its thread and replies to
it, while the post itself has no thread. Requiring the same thread cut every
comment off its post. A reply to the thread's own first message is now a link
when that message has text; a Telegram forum topic's first message is an empty
service message, so forum topics still stay apart (checked on a test store:
none of the 30 stored topic roots has text). The agent batches count a reply
as recorded by the same rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leemour
leemour merged commit dffc77e into main Nov 18, 2026
43 checks passed
@leemour
leemour deleted the fix/conversations-forum-topics branch November 18, 2026 07:54
leemour added a commit that referenced this pull request Nov 18, 2026
)

## Task
After the linking-rules version became 6 (#3107), every user's chats get
rebuilt. A rebuild of one large forum group
took about 15 minutes, most of it after the new build was already in
place.

## Why
`purgeVectorHashes` runs once per chunk text of the old build. Its
candidates query joined on
`chunks.chunkable_type`, and SQLite chose the `(chunkable_type,
chunkable_id, position)` index: every lookup read every
conversation chunk in the store instead of using
`chunks_by_content_hash`. The delete after it named only
`content_hash`, while the `embeddings` key is `(model, content_hash)`,
so it read the whole key as well.

## What
- `+k.chunkable_type` keeps the planner on `chunks_by_content_hash`
(checked with `EXPLAIN QUERY PLAN`).
- The delete runs once per stored model, by the full key. The models are
read once per purge.

## Testing
- Measured on a copy of a test store (about 220,000 conversation chunks,
two embedding models), rebuilding one forum
group of about 112,000 messages with an unchanged result, so the purge
checks about 42,600 old chunk texts:
  - before: 889 s, and 889 s again (load about 4 to 8)
  - after: 52 s (load about 4)
  - the candidates query alone: 29.5 ms per text before, 0.3 ms after.
- Full write-up and script: [cli-testing
performance/store/vector-purge](https://github.com/WireCatLabs/cli-testing/blob/perf/store-vector-purge/performance/store/vector-purge/README.md)
(WireCatLabs/cli-testing#384).
- Local: `pnpm ci:quick`, Biome, `pnpm docs:check`, vitest over
`src/store` and `src/services/embeddings`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant