Skip to content

Implement and audit the secondary-index industrial foundation - #55

Merged
forhappy merged 6 commits into
mainfrom
codex/secondary-index-industrial-foundation
Jul 30, 2026
Merged

Implement and audit the secondary-index industrial foundation#55
forhappy merged 6 commits into
mainfrom
codex/secondary-index-industrial-foundation

Conversation

@forhappy

Copy link
Copy Markdown
Contributor

Outcome

Implements the approved hard-cutover secondary-index industrial foundation and
the 2026-07-29 final architecture audit. This is the implementation follow-up
to merged planning PR #52.

The cutover adds no compatibility reader, dual publication, migration shim, or
suffix-named V2/V3 API.

Architecture and correctness

  • One canonical IndexedCollectionState root and one CAS linearization point.
  • Immutable snapshot closure binds source, physical indexes, descriptors,
    retention, and durable pins.
  • Delta state publication and checked cardinality deltas keep point mutation
    work local instead of scanning or rebuilding complete collections.
  • Persisted caps bound active indexes, retained snapshots, descriptors, and
    durable pins.
  • SQLite is the sole production-profile adapter. FileNodeStore, MemStore,
    PGlite, redb, RocksDB, and SlateDB remain explicitly verification-only.

Bounded resources

  • Incremental forward/reverse pages with charged lookahead, returned bytes,
    source fetches, retained memory, scans, and elapsed time.
  • Callback-scoped source-record joins avoid retaining an unbounded batch.
  • Streaming verification counts/diffs and aggregate verify_all budgeting.
  • Aggregate spill reader, writer, heap/live-entry, fan-in, byte, and run limits.
  • Budgeted bundle traversal, CID verification, encoding, and chunked import.

Diagnostics and release gates

  • Redacted core Debug/Display contract and stable index code/retry metadata
    through UniFFI, Kotlin, Python, Ruby, Swift, and WASM.
  • Required all-target tests, doc tests, bounded benchmark artifacts, and ASan.
  • Scheduled 1,000,000- and 10,000,000-record benchmark/stress jobs.
  • Updated API inventory and exact release evidence in
    docs/secondary-index-release-evidence.md.

Local validation

  • Rust: 501 library tests, all integration tests, 74 documentation tests.
  • Clippy with warnings denied and hard-cutover absence gate.
  • UniFFI 78/78; Node 60/60; WASM 34/34; Python 23/23; Ruby 22/22.
  • Binding inventory: 3,067 operations.
  • SQLite indexed production-profile suite passed locally.

Final audit verdict

The atomic core is strong and the implementation is materially hardened. This
PR is intentionally a draft because production certification still requires:

  1. bounded streaming CBOR bundle deserialization before allocation;
  2. operation-local physical I/O, CAS, spill, and query-budget observability;
  3. independent-process SQLite publication/reopen and forced-termination
    durability evidence;
  4. typed aggregate GC plan/sweep budgets and enforced benchmark regression
    baselines.

The required CI results must also pass on the exact PR head before release.

@forhappy
forhappy marked this pull request as ready for review July 30, 2026 00:43
@forhappy
forhappy merged commit 47a5c7d into main Jul 30, 2026
2 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant