Skip to content

docs: add docs/ARCHITECTURE.md explaining the retrieval pipeline - #56

Merged
7vignesh merged 1 commit into
7vignesh:mainfrom
wufangyong973:gfi/38-architecture-doc
Oct 13, 2026
Merged

7vignesh merged 1 commit into
7vignesh:mainfrom
wufangyong973:gfi/38-architecture-doc

Conversation

@wufangyong973

Copy link
Copy Markdown
Contributor

Closes #38.

New contributors had to reverse-engineer how retrieval works from the code and AGENTS.md. This adds a short docs/ARCHITECTURE.md that lowers that barrier, covering the scope called out in the issue:

  • The retrieval pipeline — query → three-tier FTS (AND / OR / stemmed prefix) → vector search → entity matching → weighted RRF fusion, with where each leg lives (fts.ts, vector-index.ts + embeddings.ts, entities.ts, fusion.ts, assembled in context.ts).
  • Why the FTS is three tiers, and why the IDF is applied on the query side only (so stored vectors stay a pure function of their own text).
  • The migration model — sequential named, append-only migrations in migrate.ts.
  • The thin-tools convention — logic in lib/, tools just wire it up.
  • The working agreement — measure retrieval changes with npm run bench, and the local-first no-LLM / no-network constraint.

Every claim is grounded in the existing source and its module header comments (e.g. the query-side-IDF invariant in idf.ts, the three-tier design and the stemmed-prefix +9.1 hit@5 gain in fts.ts, the weighted-RRF and re-sweep rule in fusion.ts).

Wiring: link it from README.md, and drop the now-stale "(if present)" hedge from the docs/ARCHITECTURE.md link in CONTRIBUTING.md.

New contributors had to reverse-engineer how retrieval works from the
@vercel

vercel Bot commented Oct 13, 2026

Copy link
Copy Markdown

@wufangyong973 is attempting to deploy a commit to the vignesh's projects Team on Vercel.

A member of the Team first needs to authorize it.

@7vignesh
7vignesh merged commit c1986d2 into 7vignesh:main Oct 13, 2026
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add docs/ARCHITECTURE.md explaining the retrieval pipeline and conventions

2 participants