Skip to content

feat(markdown): add CommonMark-aware extraction - #79

Open
luoxiaojian wants to merge 1 commit into
zvec-ai:mainfrom
luoxiaojian:codex/markdown-commonmark-extraction
Open

luoxiaojian wants to merge 1 commit into
zvec-ai:mainfrom
luoxiaojian:codex/markdown-commonmark-extraction

Conversation

@luoxiaojian

Copy link
Copy Markdown

Summary

Improve Markdown extraction by replacing ad-hoc heading and fence detection with CommonMark and GitHub Flavored Markdown block parsing.

  • Parse ATX and Setext headings according to CommonMark rules.
  • Respect fenced code, HTML blocks, lists, blockquotes, footnotes, and GFM tables during section discovery.
  • Prefer parsed block boundaries when splitting large Markdown documents.
  • Treat leading YAML front matter as a document envelope and exclude it from content chunks.
  • Normalize bounded top-level scalar and scalar-list front matter fields into generic frontMatter metadata.
  • Prioritize high-signal metadata such as titles, descriptions, names, and tags in embedding input.
  • Persist generic front matter metadata and bump the workspace index version.

Motivation

A YAML sequence item immediately before a closing front matter delimiter can resemble a Markdown Setext heading:

---
api_name:
- ExampleApi
---

Parse Markdown with CommonMark and GFM block rules so section discovery and chunk boundaries respect fenced code, HTML, lists, blockquotes, and tables.

Treat opening YAML front matter as a document envelope, exclude it from content chunks, and persist bounded normalized scalar metadata for retrieval.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant