Skip to content

fix: MarkdownHeaderSplitter silently drops a trailing header with no body text#12064

Draft
lntutor wants to merge 1 commit into
deepset-ai:mainfrom
lntutor:fix/markdown-splitter-trailing-header
Draft

fix: MarkdownHeaderSplitter silently drops a trailing header with no body text#12064
lntutor wants to merge 1 commit into
deepset-ai:mainfrom
lntutor:fix/markdown-splitter-trailing-header

Conversation

@lntutor

@lntutor lntutor commented Jul 19, 2026

Copy link
Copy Markdown

What's broken

With keep_headers=True (the default), MarkdownHeaderSplitter buffers a header whose only content is whitespace into pending_headers and prepends it to the next content chunk. But a header (or run of headers) at the end of the document has no next chunk, so the buffer is never flushed and the header text is dropped:

text = "# Header 1\nContent 1.\n# Header 2\n"
docs = MarkdownHeaderSplitter().run(documents=[Document(content=text)])["documents"]
"".join(d.content for d in docs)   # '# Header 1\nContent 1.\n'  -- '# Header 2' is gone

This breaks the reconstruction invariant the existing tests assert (test_basic_split, test_keep_headers_preserves_parent_headers_for_first_child both check "".join(doc.content) == original), and it's inconsistent: a middle empty header is preserved (prepended to the next chunk), a trailing one is dropped.

Fix

Track the start offset / text / level of buffered trailing headers and, before returning, flush them as a final chunk sliced from the original text (so byte-level reconstruction and downstream page counting stay exact) — mirroring how a middle empty header is already preserved.

Distinct from #11920 (content dropped before an embedded header/code-fence).

Tests

Adds test_trailing_header_without_content_is_not_dropped. Full test/components/preprocessors/test_markdown_header_splitter.py passes (35). Release note included.

With keep_headers=True (the default), a header whose only content is
whitespace is buffered into pending_headers to be prepended to the next
content chunk. A header (or run of headers) at the very end of the
document has no following chunk, so the buffer was never flushed and the
header text was dropped -- breaking the reconstruction invariant that
the split contents rejoin to the original, and inconsistent with a
middle empty header, which is preserved.

Flush any trailing buffered headers as a final chunk, sliced from the
original text so reconstruction stays exact.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N6RtoHuxrDqTUo9Mw9h4Cv
@lntutor
lntutor requested a review from a team as a code owner July 19, 2026 13:13
@lntutor
lntutor requested review from julian-risch and removed request for a team July 19, 2026 13:13
@vercel

vercel Bot commented Jul 19, 2026

Copy link
Copy Markdown

@lntutor is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@HaystackBot

Copy link
Copy Markdown
Contributor

Hi @lntutor, thanks a lot for your contribution! 🙏

We noticed that the Contributor License Agreement (CLA) check (license/cla) hasn't passed yet, so we've temporarily moved this PR to draft and paused the review assignment.

To get your PR reviewed, please sign the CLA via the link in the license/cla check below (or in the CLA bot comment). As soon as the check turns green, this PR will automatically be marked ready for review again and a reviewer will be re-assigned.

@HaystackBot
HaystackBot removed the request for review from julian-risch July 19, 2026 14:40
@HaystackBot HaystackBot added the cla-pending PR is in draft until the contributor signs the CLA label Jul 19, 2026
@HaystackBot
HaystackBot marked this pull request as draft July 19, 2026 14:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla-pending PR is in draft until the contributor signs the CLA topic:tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants