Skip to content

perf(memory): reduce local-search scan latency at 5,000 entries #2477

Backlog

Description

@spboyer

Summary

LocalMemoryStore.search reads active local entries serially on every query. A measured 5,000-entry corpus requires 5,003 reads and approximately 6.23 MB of input for one search. Investigate a focused reduction in repeated scan work without weakening governed-memory semantics or cross-process freshness.

Proposed primary owner: state-storage-dev.

Baseline evidence

Revision: 47082a81799710c3e457b594123d190c00596ebf, observed 2026-11-12. Relevant code: memory/index.ts:960-1012, with entry loading at lines 1337-1360.

Independent repeat of the source-function diagnostic:

Active entries p50/query p95/query
100 4.058 ms 7.535 ms
1,000 73.556 ms 91.553 ms
5,000 391.863 ms 442.849 ms

Fixture: 1 KiB content per entry, 10% substring hits, active governed local entries, normal audit append included, external providers disabled. At 5,000 entries, separate untimed storage instrumentation counted 5,003 reads and 6,234,647 bytes read, including config/index. The earlier baseline's p50 was 376.92 ms at that size; these are repeated baseline observations, not before/after optimization results.

Environment: Apple M4 arm64, 16 GiB, Darwin 25.6, Node 24.15.0. Three rounds of ten sequential observations per case, two untimed warmups, warm OS filesystem cache, fixture creation/imports outside timing. Cases were rotated/reversed between rounds. Production TypeScript methods were executed through Node's built-in transformation and an isolated relative-import hook.

Limit: locked dependency restoration returned a configured-feed E404, so packaged build/tests/startup were unavailable. These are local source-function diagnostics, not LLM throughput, allocation totals or application speedups. Portable fixtures and raw samples must become shared artifacts before optimization acceptance.

Scope

Parent: github/squad#2481.

One focused change to the existing local search scan/read path. No new public memory API, external providers, governance redesign, vector/semantic search, backend replacement or competing memory architecture.

Measurable acceptance and tests

For an optimization, use at least three interleaved built-baseline/change rounds with at least 30 observations per case per round; publish raw observations, per-round p50/p95, exact SHA/environment and correctly labeled CPU/I/O/memory evidence. The budgets below are proposed acceptance targets, not already-demonstrated gains. If a safe material win cannot be reproduced, record rejection/defer rather than broaden the implementation to chase the target.

  • Fixed cases: 100/1,000/5,000 active 1-KiB entries, 10% hits; add an all-miss query and a repeated query after one entry is changed. Include the normal content-free audit append.
  • Target at least 25% lower p50 for the 5,000-entry hit case than the same built baseline. Publish p95, read counts/bytes, CPU and peak/retained memory for every size; do not hide smaller-case regressions.
  • If concurrent reads are used, assert a fixed finite in-flight limit and verify read-error propagation. If retained derived state is used, test its cardinality bound and cross-process invalidation; neither mechanism is prescribed before profiling.
  • Result-identity tests preserve substring/case behavior and ordering, active/superseded/deleted filtering, class/load guidance, forbidden-query rejection and content-free audit records.
  • Freshness tests update, delete and supersede an entry and edit governing config from another process before the next query; no stale or unauthorized hit may be returned.
  • Check in the portable fixture/reproduction command and before/after results. Keep changes confined to the measured local scan plus directly dependent tests.

Related work

Link this focused measured call-site investigation to the memory/state ADR github/squad#2392; do not create a competing governance architecture. General context/token work in github/squad#884 and github/squad#1476 is distinct.

Method: dotnet/aspnetcore#70219 and microsoft/waza#846.

Activity

  1. changed the title [-]perf(memory): avoid serial full-corpus reads on every local-memory search[/-] [+]perf(memory): reduce local-search scan latency at 5,000 entries[/+] on Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions