feat: add script to preprocess and tokenize - #1

Open

mazzma12 wants to merge 17 commits into

mainfrom

Commits on Mar 9, 2026

wip
mazzma12
committed
5cc7b31
handle jsonl and modify build make
mazzma12
committed
04239f6
add -mode flag
mazzma12
committed
d527d1f

Commits on Mar 10, 2026

remove single and double quotes
mazzma12
committed
3d908b6
lexer: handle multiline comment after keyword
blotus
committed
59decef
wip
mazzma12
committed
24c5dbb
handle jsonl and modify build make
mazzma12
committed
743ac5e
add -mode flag
mazzma12
committed
2689efd
remove single and double quotes
mazzma12
committed
7d6f3ef
Feat: remove SPACE from model
mazzma12
committed
03e6e43

Commits on Mar 11, 2026

Revert "Feat: remove SPACE from model"
mazzma12
committed
d54a327

Commits on Mar 13, 2026

Commits on Mar 24, 2026

handle newlines in queries
blotus
committed
12baa9a

Commits on Aug 6, 2026

merge results : accepts new tests
mazzma12
committed
5564b21