Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 26 additions & 24 deletions STATUS.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,30 +12,32 @@ Baseline for the Mamba-3 RAG assistant. Update this file when a step moves, and

## Changelog

- **2026-11-09** — v2 rerank is implemented locally on `boss_ranker`. `retrieve()` rescores the top 20 hybrid hits with `Qwen/Qwen3-Reranker-0.6B`. `score` is the probability of yes, and `timing_ms.rerank` is that stage. Step 6 moves from 67% to 100%. Weighted total recomputed from the step table is 66.3%. Not on main.
- **2026-11-08** — v1 hybrid retrieval is implemented locally on `boss_ranker`: in-process BM25 and exact cosine, up to 50 each, merged with reciprocal rank fusion (rank constant 60). `index/embed.py` fills `chunks.embedding` and `embedding_model` with `Qwen/Qwen3-Embedding-0.6B` (1024-d). Query text uses the instruction wrapper; documents are stored chunk text. Null embeddings stay on the keyword list. v2 rerank is not started. Step 5 moves from 25% to 80%. Step 6 moves from 33% to 67%. Weighted total recomputed from the step table is 61.4%. Not on main.
- **2026-11-08** — v0 retrieval is implemented locally: in-process BM25 (`k1=1.2`, `b=0.75`) and the code tokenizer in `retrieval/`. `retrieve()` returns the retrieve contract; embed and rerank timings are 0. v1 hybrid and v2 rerank are not started. Step 6 moves from 8% to 33% (the lexical version only). Weighted total recomputed from the step table is 46.9%. Not on main.
- **2026-11-08** — Retrieval decision record replaced in `retrieval/README.md`. v0 is in-process BM25 with a code tokenizer; Postgres `ts_rank` is not the baseline. v1 is reciprocal rank fusion with `Qwen/Qwen3-Embedding-0.6B` at 1024 dimensions. v2 rescores the top 20 with `Qwen/Qwen3-Reranker-0.6B`. `index/schema.sql` is `vector(1024)` and no longer creates an IVFFlat index. Not implemented. Step 6 stays 8%.
- **2026-11-08** — Retrieval plan locked in `retrieval/README.md`. First version is Postgres `ts_rank`. BM25 is a later ranker on the same function. Then hybrid with reciprocal rank fusion (embedder still unset), then `cross-encoder/ms-marco-MiniLM-L-6-v2` over 20 candidates. Not implemented yet.
- **2026-11-08** — Baseline. Mamba-3 can generate and ingest can write `chunks.jsonl`. Nothing retrieves those chunks and feeds them to Mamba-3. `rag_cli.py` is a separate keyword prototype that calls Mamba-130M, not Mamba-3.

## Team update (Nov 8)
## Team update (Nov 9)

We can load a Mamba-3 SISO checkpoint and generate text, and we can crawl, filter, secret-scan, and chunk repos into `chunks.jsonl`. That ingest path is on main, with tests and CI.
`retrieve()` on `boss_ranker` now rescores its top 20 hybrid hits with `Qwen/Qwen3-Reranker-0.6B`. The returned score is that reranker's yes-probability. BM25 and cosine are the first stage. This is local, not on main.

We cannot yet ask a question and get a Mamba-3 answer grounded in CDS code. Embeddings are not generated. Local v0 retrieval ranks loaded chunk text with BM25; hybrid and rerank are not started. `/generate`, prompting, eval, and the app are still stubs. Local uncommitted work also adds Postgres, a text-only chunk loader, and `rag_cli.py`. That CLI does not use the ingest chunks or the Mamba-3 package.
We still cannot ask a question and get a Mamba-3 answer grounded in CDS code. The embed script has not been run on a corpus. Prompting, eval, and the app are still stubs.

Weighted total from the step table is 46.9%. The generator is 80% in that table. Retrieved chunks are not yet fed to Mamba-3.
Weighted total from the step table is 66.3%. Retrieved chunks are not yet fed to Mamba-3.

## Steps

Weights are the share of a functional assistant (question in, cited Mamba-3 answer out), not lines of code. Weighted total of the Done column: **46.9%**.
Weights are the share of a functional assistant (question in, cited Mamba-3 answer out), not lines of code. Weighted total of the Done column: **66.3%**.

| Step | What | Status | Done | Weight |
|---|---|---|---|---|
| Model | Mamba-3 load and generate | Partial | 80% | 20 |
| 2 | Postgres + pgvector schema | Partial, local only | 70% | 5 |
| 3–4 | Ingest: crawl, filter, scan, chunk | Done | 90% | 18 |
| 5 | Embed and store | Partial, local only | 25% | 17 |
| 6 | `retrieve()`: BM25, then hybrid, then rerank | Partial, local only | 33% | 15 |
| 5 | Embed and store | Partial, local only | 80% | 17 |
| 6 | `retrieve()`: BM25, then hybrid, then rerank | Partial, local only | 100% | 15 |
| 7 | `/generate` around Mamba-3 | Not started | 15% | 10 |
| 8 | Prompt + citation mapping | Not started | 10% | 5 |
| 9 | App: `/ask` + chat page | Not started | 0% | 5 |
Expand All @@ -54,9 +56,9 @@ On main.

### Step 2 — Postgres schema (70%, local only)

Not on main. Untracked: `docker-compose.yml`, `index/schema.sql`, `index/test_db.py`.
`index/schema.sql` is on main and safe to re-run. Still untracked: `docker-compose.yml`, `index/load_chunks.py`, `index/test_db.py`.

- Schema matches `contracts/chunk.schema.json`: chunk metadata, `embedding vector(1024)` for `Qwen/Qwen3-Embedding-0.6B`, `tsvector`, trigram index. No ANN index at init. The `english` `tsvector` trigger remains; retrieval will not call it.
- Schema matches `contracts/chunk.schema.json`: chunk metadata, `embedding vector(1024)` for `Qwen/Qwen3-Embedding-0.6B`, `embedding_model` (both null or both set), `simple` tsvector, trigram index. No ANN index at init. Retrieval does not call the tsvector trigger.
- `psycopg` is a project dependency (retrieval uses it too). The loader is still local only.

### Steps 3–4 — Ingest (90%)
Expand All @@ -72,22 +74,22 @@ On main. This is the piece that moved in PRs #2–#5.
- `samples/chunks.jsonl` is 100 chunks for retrieval work.
- Still open: a full org crawl checked into nothing (output belongs in gitignored `data/`), and the root `README.md` still says the repo is only the model setup.

### Step 5 — Embed and store (25%, local only)
### Step 5 — Embed and store (80%, local only)

Store half is local. Embed half is not written.
The loader and the embed job are both written. Not on main. No corpus has been embedded yet.

- `index/load_chunks.py` upserts chunk text and metadata. It never writes `embedding`.
- `index/README.md` lists `embed.py` as TODO. That file does not exist.
- Embedding model is locked in `retrieval/README.md`: `Qwen/Qwen3-Embedding-0.6B`, 1024 dimensions, cosine. The column is `vector(1024)`. Nothing has been embedded.
- `index/load_chunks.py` upserts chunk text and metadata. It does not write `embedding`. It is on this branch, not on main.
- `index/embed.py` fills `embedding` and `embedding_model` for rows where `embedding` is null. Model is `Qwen/Qwen3-Embedding-0.6B`, 1024 dimensions, L2-normalized. `--force` re-embeds every row. The schema check requires the vector and the model name together.
- Exact cosine until the table is large. No HNSW index.

### Step 6 — `retrieve()` (33%, local only)
### Step 6 — `retrieve()` (100%, local only)

v0 BM25 is implemented locally. v1 hybrid and v2 rerank are not started. 33% is one of the three locked versions, not a claim that hybrid or rerank exists.
v0 BM25, v1 hybrid, and v2 rerank are implemented locally. 100% is all three locked versions. Not on main.

- `retrieve(question, k=4, repos=None)` in `retrieval/` returns `contracts/retrieve.schema.json`. No HTTP route. Tests inject chunks or a fake connection, so they do not need Postgres. Not on main.
- v0: in-process BM25 (`k1=1.2`, `b=0.75`) over chunk text, with the code tokenizer (lowercase, split on non-letters, split camelCase / PascalCase / snake_case, keep the raw identifier). `score` is the BM25 score. `timing_ms.embed` and `timing_ms.rerank` are 0. `timing_ms.search` is this pass. Postgres `ts_rank` and the `english` tsvector are not used.
- v1: hybrid. Not started. Keyword and vector lists, up to 50 each, merged by reciprocal rank fusion. Embedder is `Qwen/Qwen3-Embedding-0.6B` (1024-d). Exact cosine until the table is large.
- v2: rerank. Not started. Rescore the top 20 hybrid hits with `Qwen/Qwen3-Reranker-0.6B`. `score` becomes the reranker score.
- `retrieve(question, k=4, repos=None)` in `retrieval/` returns `contracts/retrieve.schema.json`. No HTTP route. Tests inject chunks, a fake connection, a fake embedder, or a fake reranker, so they do not need Postgres or a model download.
- v0: in-process BM25 (`k1=1.2`, `b=0.75`) over chunk text, with the code tokenizer (lowercase, split on non-letters, split camelCase / PascalCase / snake_case, keep the raw identifier). This is the keyword list. Postgres `ts_rank` and the `simple` tsvector are not used.
- v1: hybrid. Keyword and vector lists, up to 50 each, merged by reciprocal rank fusion (`1 / (60 + rank)`, rank starting at 1). Embedder is `Qwen/Qwen3-Embedding-0.6B` (1024-d). Exact cosine in process. A null embedding stays on the keyword list only. With no embeddings, the first stage follows BM25 and `timing_ms.embed` is 0. Otherwise `timing_ms.embed` is the query embed and `timing_ms.search` is the two lists plus the merge.
- v2: rerank. Rescore the top 20 first-stage hits with `Qwen/Qwen3-Reranker-0.6B`. `score` is the probability of yes. `timing_ms.rerank` is this stage. The response is at most `k` of those 20. Equal scores keep the first-stage order.
- `rag_cli.py` (untracked) is not this step. It does not read the chunks table or the contract.

### Step 7 — `/generate` (15%)
Expand Down Expand Up @@ -121,15 +123,15 @@ On main, and the thing later steps should code against:

Untracked experiments and notes, separate from the steps above: `mamba.py/`, `mamba-130m.mlx.npz`, `test_mamba.py`, `ADVISOR_PRESENTATION.md`, `ARCHITECTURE_DEEP_DIVE.md`, `IMPLEMENTATION_DETAILS.md`, `POSTGRES_SETUP.md`, `RETRIEVAL_ARCHITECTURE.md`, `RETRIEVAL_OPTIONS_FOR_ADVISOR.md`, `.claude/`.

`ADVISOR_PRESENTATION.md` (October) marks Postgres, the loader, and chunking as complete infrastructure. Chunking is on main. Postgres and the loader are still local, and embeddings are not implemented. Use this file, not that deck, for current status.
`ADVISOR_PRESENTATION.md` (October) marks Postgres, the loader, and chunking as complete infrastructure. Chunking and `index/schema.sql` are on main. The loader is still local. The embed job, hybrid first stage, and rerank are local and have not been run on a corpus. Use this file, not that deck, for current status.

## Shortest path from here

1. v0 `retrieve()` is implemented locally (in-process BM25, retrieve-contract response, no embeddings). Next retrieval work is v1 hybrid, which needs embeddings.
2. Add `embed.py`. Fill `chunks.embedding` (1024-d, `Qwen/Qwen3-Embedding-0.6B`) for rows already loaded, then v1 hybrid.
1. Load a JSONL into Postgres, then run `uv run python index/embed.py` so `chunks.embedding` is filled.
2. Ask `retrieve()` a question against those rows.
3. Build the numbered-source prompt from those chunks.
4. Generate with `mamba3.load_model` (SISO checkpoint, Llama-3.1 tokenizer).
5. Run one real corpus: pipeline or chunker, load the JSONL, ask a question.
5. Run one real corpus: pipeline or chunker, load the JSONL, embed, ask a question.

After that loop works: v2 rerank of the top 20, the `/generate` service, citation checking, the 50-question set, and the `/ask` app. Domain-adaptive training, the CUDA kernels, and a Transformer baseline are the research comparison, not blockers for the first answer.

Expand Down
24 changes: 24 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
services:
postgres:
image: pgvector/pgvector:pg17
container_name: adhd-postgres
environment:
POSTGRES_DB: adhd_rag
POSTGRES_USER: adhd
POSTGRES_PASSWORD: dev_password_change_in_prod
POSTGRES_HOST_AUTH_METHOD: trust # For local dev only
ports:
- "5432:5432"
volumes:
- postgres_data:/var/lib/postgresql/data
- ./index/schema.sql:/docker-entrypoint-initdb.d/01-schema.sql
healthcheck:
test: ["CMD-SHELL", "pg_isready -U adhd -d adhd_rag"]
interval: 5s
timeout: 5s
retries: 5
restart: unless-stopped

volumes:
postgres_data:
driver: local
11 changes: 11 additions & 0 deletions index/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,14 @@ index/setup_local.sh
connects to by default, then applies `schema.sql`. Both are safe to re-run. A database created
from an older `schema.sql` keeps its English-stemmed `text_search` values until rows are rewritten;
drop and recreate it for a clean start.

## Embed

`index/embed.py` writes `embedding` and `embedding_model` together (`Qwen/Qwen3-Embedding-0.6B`, 1024 dimensions, L2-normalized). The schema check rejects a vector without a model name. By default only rows with a null embedding are embedded. `--force` re-embeds every row.

```bash
uv run python index/embed.py
uv run python index/embed.py --force --batch-size 8
```

Load chunk text first (`index/load_chunks.py`), then embed. `retrieve()` reads the vectors. Chunks with a null embedding stay on the BM25 list only.
85 changes: 85 additions & 0 deletions index/embed.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
"""Fill chunks.embedding for rows that do not have a vector yet.

The schema requires embedding and embedding_model to be set together.
Documents are embedded as stored chunk text (no query instruction).

Usage:
uv run python index/embed.py
uv run python index/embed.py --force --batch-size 8
"""

import argparse
import sys

import psycopg
from pgvector import Vector
from pgvector.psycopg import register_vector

from retrieval.embed import DIMENSIONS, MODEL_ID, embed_documents
from retrieval.retrieve import resolve_database_url


def write_embeddings(database_url: str, *, force: bool = False, batch_size: int = 8) -> int:
"""Write 1024-d vectors. Returns how many rows were updated.

By default only rows with a null embedding are embedded. ``force``
re-embeds every row and replaces ``embedding_model``.
"""
if batch_size < 1:
raise ValueError("batch_size must be at least 1")

conn = psycopg.connect(database_url)
try:
register_vector(conn)
sql = "SELECT id, text FROM chunks"
if not force:
sql += " WHERE embedding IS NULL"
sql += " ORDER BY id"
with conn.cursor() as cur:
cur.execute(sql)
rows = cur.fetchall()
if not rows:
return 0

written = 0
for start in range(0, len(rows), batch_size):
batch = rows[start : start + batch_size]
vectors = embed_documents([text for _chunk_id, text in batch])
with conn.cursor() as cur:
for (chunk_id, _text), vector in zip(batch, vectors, strict=True):
if len(vector) != DIMENSIONS:
raise ValueError(f"{MODEL_ID} returned {len(vector)} dimensions, expected {DIMENSIONS}")
cur.execute(
"""
UPDATE chunks
SET embedding = %s, embedding_model = %s
WHERE id = %s
""",
(Vector(vector), MODEL_ID, chunk_id),
)
conn.commit()
written += len(batch)
return written
finally:
conn.close()


def main():
parser = argparse.ArgumentParser(description="Embed chunks with Qwen/Qwen3-Embedding-0.6B")
parser.add_argument("--force", action="store_true", help="Re-embed every row, not only null embeddings")
parser.add_argument("--batch-size", type=int, default=8, help="Chunks per forward pass (default: 8)")
parser.add_argument("--db-url", default=None, help="Postgres URL (default: DATABASE_URL or the local adhd_rag URL)")
args = parser.parse_args()

database_url = resolve_database_url(args.db_url)
try:
written = write_embeddings(database_url, force=args.force, batch_size=args.batch_size)
except psycopg.OperationalError as exc:
print(f"Failed to connect to database: {exc}", file=sys.stderr)
print("Apply index/schema.sql first (index/setup_local.sh, or docker compose up -d).", file=sys.stderr)
sys.exit(1)
print(f"Embedded {written} chunks with {MODEL_ID}")


if __name__ == "__main__":
main()
Loading