Skip to content

RESULTS.md

Latest commit

 

History

History
46 lines (33 loc) · 2.44 KB

RESULTS.md

File metadata and controls

46 lines (33 loc) · 2.44 KB
Raw

Results

This file separates what is verified today from what requires a live run, so nothing here is overstated.

Verified now — metric behavior (no external stack)

The custom metrics are unit-tested deterministically (make test, 28 metric tests + 6 chunker tests). These prove the behavior the project claims, independent of any LLM:

Claim Test evidence
Overlapping chunks don't over-penalize precision section_aware_precision groups 3 overlapping windows on one table into 1 relevant unit (0.5, not 0.25)
Redundancy can't lower recall (the #2788 fix) section_aware_recall is monotonic when a redundant chunk is added
Section routing surfaces the right statement balance-sheet chunk out-ranks a higher-similarity risk-factors chunk for an "assets" query; no-op without intent
Multi-hop scoring finds the weak hop R&D%-of-revenue question flags the failing revenue hop, not the R&D hop
Table extraction quality catches dropped rows 2/5 extracted cells → score 0.40, missing cells named
Confident wrong numbers are flagged asserting "$14.2B R&D" against a "$31.4B" context → unsupported figure flagged
Weak metrics map to config actions low precision/recall/faithfulness → concrete --deduplicate / top_k / temperature recommendations

Run it yourself:

make dev && make test

Requires a live run — end-to-end retrieval scores

Full pipeline scores (retrieval + llama3 generation + DeepEval judge) need the local stack up (Docker + Ollama). They are intentionally not filled in here rather than estimated. To generate them:

make db-up && make db-init
make download                 # Apple FY2024 + FY2025 10-Ks
make ingest FILE=<filing>     # per downloaded filing
make ablation                 # raw / dedup / routing, stored in eval_runs

Then paste the summary tables below.

End-to-end scores (fill after a live run)

Config ctx precision ctx recall faithfulness section precision section recall numeric grounding
raw tbd tbd tbd tbd tbd tbd
+ dedup tbd tbd tbd tbd tbd tbd
+ routing tbd tbd tbd tbd tbd tbd

The initial manual finding (2/3 honest refusals, 1/3 confident hallucination) is documented in notes and the README.