Skip to content

Add production async proximity map lifecycle - #63

Merged
forhappy merged 2 commits into
mainfrom
codex/async-proximity-map
Aug 3, 2026
Merged

Add production async proximity map lifecycle#63
forhappy merged 2 commits into
mainfrom
codex/async-proximity-map

Conversation

@forhappy

@forhappy forhappy commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

What changed

  • add the complete native AsyncProximityMap lifecycle over AsyncStore: canonical build/load, reads, scans, mutation/rebuild, verification, native search, membership proofs, structural proofs, and search proofs
  • share deterministic localized copy-on-write hierarchy mutation between sync and async execution instead of rebuilding the full async hierarchy for ordinary vector edits
  • add descriptor-derived authenticated search I/O binding, cache-aware batched reads, request coalescing, external-vector batching, and cancellation-safe in-flight cleanup
  • add AsyncProximityHead for validated named opens, build/publish-if-absent, shared runtime reuse, expected-head CAS, and bounded conflict retry
  • reuse authenticated closure validation for unchanged managed heads and carry validation through managed copy-on-write mutations
  • add async-only HNSW/PQ construction, bounded catalog publication, composite maintenance, and catalog binding
  • add explicit async build record/memory/CPU/publication limits and stream the sorted exact-directory publication
  • add PostgreSQL 16 end-to-end and performance coverage at 1,836 dimensions

Why

The synchronous proximity implementation already had localized immutable mutation, but the async implementation rebuilt all proximity routing whenever vectors changed. Remote applications also had to compose decoder context, caching, typed-root validation, publication, CAS, retry, and accelerator maintenance themselves. Those shallow seams amplified remote round trips and made correct service integration unnecessarily difficult.

This change makes the async path a first-class service interface while preserving canonical byte/CID equivalence with the synchronous implementation and clean-rebuild oracle.

Developer impact

  • ordinary async vector mutation now rewrites affected paths and clusters only
  • load_with_runtime and load_with_search_io derive decoder context from the authenticated descriptor
  • managed heads reduce repeated validated-open cost without weakening raw untrusted-load validation
  • remote-only applications can construct and maintain HNSW, PQ, composite, and accelerator catalogs without a synchronous remote adapter
  • explicit resource and provider-batch limits make large remote builds predictable

Validation

  • cargo test --features async-store --lib — 502 passed
  • cargo test --features async-store --doc — 74 passed
  • focused async, mutation, overflow, quantization, and proof suites — 33 passed
  • cargo clippy --features async-store --lib -- -D warnings
  • PostgreSQL adapter test clippy with warnings denied
  • cargo +1.89.0 check --features async-store
  • public interface inventory check — 3,213 operations
  • repeated PostgreSQL 16 release-mode E2E at 1,000 records × 1,836 dimensions

Three-run PostgreSQL medians:

Operation Median
Whole-map build 509.4 ms
Whole-map verification 626.7 ms
Cold exact search 114.1 ms/query
Warm exact search 1.915 ms/query
Localized vector mutation 95.9 ms
Clean rebuild oracle 432.3 ms

Localized vector mutation was 4.5× faster than the same-run clean rebuild and used 22 SQL calls. In a representative managed-head run, cached open took 3.2 ms and 2 SQL calls versus 431.7 ms and 144 calls for a generic fully validated named-root load.

Build and verification values are totals for the complete 1,000-record map, not per-record latency.

Known upstream baseline

The repository-wide cargo test --features async-store command still encounters pre-existing stale secondary-index imports in tests/async_store.rs on main. Library, documentation, focused proximity, PostgreSQL E2E, clippy, and MSRV checks above pass.

@forhappy
forhappy marked this pull request as ready for review August 3, 2026 07:37
@forhappy
forhappy merged commit da44727 into main Aug 3, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant