Skip to content

Scalability: journal:index is read/rewritten in full on every append and every list #3

Description

@dkritarth

src/kv.ts appendJournalEntry reads the whole journal:index array, pushes one id, writes the whole array back on every single append. listJournalEntries reads journal:index then fetches every entry with kv.get in a Promise.all, then filters by since/limit in memory - no pagination against KV itself.

As the journal grows this is O(n) KV read+write per append and O(n) KV reads per list call. Since there is no promotion-driven pruning of old entries visible in scripts/promote.ts (it never deletes, only tracks review state locally), journal:index will grow unbounded over the life of the deployment.

Worth deciding: either (a) archive/prune promoted entries out of journal:index via a script, or (b) switch listing to KV list-by-prefix with cursor pagination instead of an index array, or (c) explicitly document a expected max journal size and revisit before it is exceeded.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions