feat(index): HNSW writer bridge for Pebble crawl path + ops systemd units#36
Merged
TeoSlayer merged 1 commit intoJul 23, 2026
Conversation
…md units internal/index/hnsw_writer.go implements PassageWriter against a Pebble store + in-memory HNSW graph, with periodic checkpointing and a manual Flush for shutdown. Covered by hnsw_writer_test.go (bridges passages into the graph, and verifies the auto-flush-at-threshold contract). Not yet wired into cmd_crawl/cmd_serve — those already use their own hnswPassageWriter in cmd/cosift; this is an alternate, independently tested implementation living in internal/index for review. Also adds the systemd unit/timer files used to run cosift-crawl, cosift-serve, cosift-snapshot, and the ollama service(s) in production. No secrets included — cosift-snapshot.service loads its admin token via EnvironmentFile from /etc/cosift/snapshot.env, not inline. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
TeoSlayer
merged commit Jul 23, 2026
3daf95b
into
fix/warn-high-crawl-embed-concurrency
3 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
internal/index/hnsw_writer.go, aPassageWriterimplementation that bridges passages into an in-memory HNSW graph and periodically persists to astore.PebbleStore(checkpoint-on-threshold + manualFlush). Covered byinternal/index/hnsw_writer_test.go.cmd_crawl/cmd_serve— those currently use their ownhnswPassageWriterdefined incmd/cosift. This is an independently tested, standalone implementation added for review; wiring/dedup is a follow-up decision, not done here.cosift-crawl.service/.timer,cosift-serve.service,cosift-snapshot.service/.timer,ollama.service,ollama-replica.service.tmpl. Checked for secrets — the snapshot service loads its admin token viaEnvironmentFile=-/etc/cosift/snapshot.env, nothing is inlined.What's intentionally excluded
cmd/cosift/pebble_serve.gowas found causing duplicate declarations againstserve_admin.go/serve_answer.go/serve_search.go(go build ./...failed). It was removed locally (never tracked, so nothing to commit) — this is what makes the base branch build again.output/,paper/,tmp/scratch dirs (~49MB, LaTeX/PDF scratch) anddocs/STARVATION-REQUIREMENTS.md(stale, unrelated draft) were left untouched and not staged.Test plan
go build ./...passesgo test ./internal/index/...passes🤖 Generated with Claude Code