fix: MarkdownHeaderSplitter silently drops a trailing header with no body text#12064
fix: MarkdownHeaderSplitter silently drops a trailing header with no body text#12064lntutor wants to merge 1 commit into
Conversation
With keep_headers=True (the default), a header whose only content is whitespace is buffered into pending_headers to be prepended to the next content chunk. A header (or run of headers) at the very end of the document has no following chunk, so the buffer was never flushed and the header text was dropped -- breaking the reconstruction invariant that the split contents rejoin to the original, and inconsistent with a middle empty header, which is preserved. Flush any trailing buffered headers as a final chunk, sliced from the original text so reconstruction stays exact. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N6RtoHuxrDqTUo9Mw9h4Cv
|
@lntutor is attempting to deploy a commit to the deepset Team on Vercel. A member of the Team first needs to authorize it. |
|
|
|
Hi @lntutor, thanks a lot for your contribution! 🙏 We noticed that the Contributor License Agreement (CLA) check ( To get your PR reviewed, please sign the CLA via the link in the |
What's broken
With
keep_headers=True(the default),MarkdownHeaderSplitterbuffers a header whose only content is whitespace intopending_headersand prepends it to the next content chunk. But a header (or run of headers) at the end of the document has no next chunk, so the buffer is never flushed and the header text is dropped:This breaks the reconstruction invariant the existing tests assert (
test_basic_split,test_keep_headers_preserves_parent_headers_for_first_childboth check"".join(doc.content) == original), and it's inconsistent: a middle empty header is preserved (prepended to the next chunk), a trailing one is dropped.Fix
Track the start offset / text / level of buffered trailing headers and, before returning, flush them as a final chunk sliced from the original text (so byte-level reconstruction and downstream page counting stay exact) — mirroring how a middle empty header is already preserved.
Distinct from #11920 (content dropped before an embedded header/code-fence).
Tests
Adds
test_trailing_header_without_content_is_not_dropped. Fulltest/components/preprocessors/test_markdown_header_splitter.pypasses (35). Release note included.