0.2.15 - #35
Merged
Merged
Conversation
…PIPELINE_MAP, FACULTY_MAP, DOC_MAP) [skip ci]
…PIPELINE_MAP, FACULTY_MAP, DOC_MAP) [skip ci]
…x N) The term-major postings loop probed every (term, doc) pair whether or not the term occurs in the doc. On real prose vocab grows with N, so the build was effectively superlinear in corpus size: at BEIR NQ scale (2,681,468 docs, vocab 821,276 under this file's own tokenize) that is 2.2e12 probes and the build does not complete. The doc-major reorder -- one pass over each doc's term counts, appending to per-term lists -- is O(total tokens) and finished NQ in 309.8 s including tokenization. The postings are identical by construction: same idf, same per-(term, doc) weight expression with the same operands (same IEEE bits), same ascending doc order per term. tests/test_bm25_docmajor_build.py rebuilds the postings with the original term-major loop verbatim and asserts np.array_equal on every term's index and weight arrays, plus scores() == _scores_reference(). The existing _selftest's bit-identity assertion passes unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…eak-free input cannot lose facts chunk_text's docstring names the failure mode it exists to avoid -- a fact split across two chunks is retrievable from neither -- but the fallback for one oversized paragraph was plain p[:max_chars] windows: fixed and non-overlapping. Input with no blank lines at all (a pasted log, a minified file, a NIAH haystack) is ONE runaway paragraph, so the whole document took that path, and any fact straddling a max_chars boundary survived in no chunk. Measured (600-char chunks, 86 needle offsets, production harness): blob input lost 8/86 offsets (9%); the same prose with paragraph breaks lost 0/86. Fix: the degenerate path now strides max_chars - overlap (overlap clamped to max_chars // 2 so the stride stays positive), which makes any fact shorter than overlap unloseable. The paragraph path is byte-identical to before; overlap=0 restores the old windows exactly. tests/test_knowledgestore_chunk_overlap.py sweeps a needle across 86 offsets in the same text presented as blob and as paragraphs, asserts zero losses on both arms, pins the paragraph path's independence from overlap, and keeps the old failure reproducible at overlap=0 so the sweep can never go soft. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…PIPELINE_MAP, FACULTY_MAP, DOC_MAP) [skip ci]
…08-14 Two measured fixes (O(tokens) BM25 build, chunk_text overlap) + bench-campaign findings
…PIPELINE_MAP, FACULTY_MAP, DOC_MAP) [skip ci]
…PIPELINE_MAP, FACULTY_MAP, DOC_MAP) [skip ci]
…PIPELINE_MAP, FACULTY_MAP, DOC_MAP) [skip ci]
…PIPELINE_MAP, FACULTY_MAP, DOC_MAP) [skip ci]
…PIPELINE_MAP, FACULTY_MAP, DOC_MAP) [skip ci]
…PIPELINE_MAP, FACULTY_MAP, DOC_MAP) [skip ci]
…PIPELINE_MAP, FACULTY_MAP, DOC_MAP) [skip ci]
…PIPELINE_MAP, FACULTY_MAP, DOC_MAP) [skip ci]
…PIPELINE_MAP, FACULTY_MAP, DOC_MAP) [skip ci]
…PIPELINE_MAP, FACULTY_MAP, DOC_MAP) [skip ci]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.