fix(crawler): warn when COSIFT_CRAWL_EMBED_CONCURRENCY can starve /search#34
Open
TeoSlayer wants to merge 2 commits into
Open
fix(crawler): warn when COSIFT_CRAWL_EMBED_CONCURRENCY can starve /search#34TeoSlayer wants to merge 2 commits into
TeoSlayer wants to merge 2 commits into
Conversation
…arch The crawler embedder is throttled (default cap 8) so bulk-crawl embeds can't starve interactive /search. The COSIFT_CRAWL_EMBED_CONCURRENCY override accepted any value with no guard, so setting it far above the safe range silently defeats that protection — in prod a stale value of 256 dropped /search from ~2s to >60s. Emit a loud startup WARNING when the cap exceeds a safe ceiling (OLLAMA_NUM_PARALLEL when set, else 4x the default). No clamp: the operator keeps control. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
…md units (#36) internal/index/hnsw_writer.go implements PassageWriter against a Pebble store + in-memory HNSW graph, with periodic checkpointing and a manual Flush for shutdown. Covered by hnsw_writer_test.go (bridges passages into the graph, and verifies the auto-flush-at-threshold contract). Not yet wired into cmd_crawl/cmd_serve — those already use their own hnswPassageWriter in cmd/cosift; this is an alternate, independently tested implementation living in internal/index for review. Also adds the systemd unit/timer files used to run cosift-crawl, cosift-serve, cosift-snapshot, and the ollama service(s) in production. No secrets included — cosift-snapshot.service loads its admin token via EnvironmentFile from /etc/cosift/snapshot.env, not inline. Co-authored-by: Teodor Calin <teodor@vulturelabs.io> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The crawler embedder is throttled (default cap 8) so bulk-crawl embeds can't starve interactive
/search. TheCOSIFT_CRAWL_EMBED_CONCURRENCYoverride accepted any positive value with no guard, so setting it well above the safe range silently defeats that protection.In production this bit us: the var had drifted to 256 (a backfill leftover), which dropped
/searchfrom ~2s to >60s — the crawler's concurrent embed→insert load starved search traversal on the HNSW graph. It was a deep debug precisely because nothing flagged the misconfiguration.This adds a single startup WARNING when the cap exceeds a safe ceiling (
OLLAMA_NUM_PARALLELwhen set, else 4× the default). It does not clamp — an operator on a large box may legitimately want more; they just get told.No behavior change, no docs.
🤖 Generated with Claude Code