From 7b2fd19da191e7cb2dc35e294abb6034513e3938 Mon Sep 17 00:00:00 2001 From: gspark Date: Sat, 24 Oct 2026 05:23:29 +0900 Subject: [PATCH 1/7] test: freeze four-model held-out structured answer evaluation (#263) --- opt_prime/examples/pd/http_matrix/README.md | 26 ++ opt_prime/examples/pd/http_matrix/STATUS.md | 4 + .../evidence263/fixture-manifest.json | 74 ++++++ .../evidence263/quality263-full.log | 46 ++++ .../quality263-gemma-fixtures.json.gz | Bin 0 -> 23984 bytes .../evidence263/quality263-http-red.log | 48 ++++ .../quality263-koni-fixtures.json.gz | Bin 0 -> 31333 bytes .../quality263-qwen06b-fixtures.json.gz | Bin 0 -> 24169 bytes .../quality263-qwen4b-fixtures.json.gz | Bin 0 -> 24108 bytes .../evidence263/research-events.jsonl | 2 + .../pd/http_matrix/evidence263/tests-red.log | 237 ++++++++++++++++++ .../2026-10-24-heldout-quality263.md | 42 ++++ .../pd/http_matrix/heldout_quality.py | 133 ++++++++++ opt_prime/tests/test_heldout_quality.py | 92 +++++++ 14 files changed, 704 insertions(+) create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/fixture-manifest.json create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-full.log create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-fixtures.json.gz create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-http-red.log create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-fixtures.json.gz create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen06b-fixtures.json.gz create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen4b-fixtures.json.gz create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/research-events.jsonl create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/tests-red.log create mode 100644 opt_prime/examples/pd/http_matrix/experiments/2026-10-24-heldout-quality263.md create mode 100644 opt_prime/examples/pd/http_matrix/heldout_quality.py create mode 100644 opt_prime/tests/test_heldout_quality.py diff --git a/opt_prime/examples/pd/http_matrix/README.md b/opt_prime/examples/pd/http_matrix/README.md index b260e35b..2c243a39 100644 --- a/opt_prime/examples/pd/http_matrix/README.md +++ b/opt_prime/examples/pd/http_matrix/README.md @@ -51,3 +51,29 @@ Run from the repository root. Omit the producer arguments for native control. `p Subprocess producers expose a boolean `ready` property; pools require every configured member that implements it to be ready. Batched `/ready` and `/health` return503 after a known terminal failure/close, and new completions receive retryable503 before admission. A healthy replacement at an idle boundary restores availability without poisoning the decoder. Individual request errors do not make a still-running producer unready. No pool member is silently removed and no request is automatically retried. Workers without this optional property retain prior behavior. Existing requests keep their existing completion/error handling. See producer-readiness246 for the real subprocess/HTTP reproduction and regression results. + +### Held-out structured answers (#263) + +`heldout_quality.py` freezes 36 synthetic cases per model: four addition, +passkey, and Korean table variants at short/~2K/~8K lengths. It renders the +model's chat template with thinking disabled, then sends the same input IDs +to each route. Inputs and hashes are frozen in `evidence263/` before inference. +Decompress each `*-fixtures.json.gz` with `gzip -dk` to reuse the exact inputs. + +```bash +python heldout_quality.py run --fixtures INPUT.json --endpoint http://localhost:8000 \ + --route native-gpu --output RESULTS.jsonl +``` + +Use normal EOS, greedy sampling, output cap128, no prefix/host caching. The +runner refuses output overwrite, retains raw text/token IDs, and stops on +an HTTP/inference failure. Start the server using the same runtime/model +configuration being compared; the runner does not launch or alter it. + +`answer_correct` requires an exact string answer in a parsed JSON object; +`format_correct` separately requires exactly one `answer` key. Duplicate keys, +non-string answers, trailing prose and malformed JSON fail. Thus this measures +structured-answer adherence, not unrestricted semantic quality. `compare` +requires matching case sets and input hashes and reports both correct→wrong +and wrong→correct transitions plus exact token matches. Do not erase existing +strict token failures or infer general equivalence from these 36 synthetic cases. diff --git a/opt_prime/examples/pd/http_matrix/STATUS.md b/opt_prime/examples/pd/http_matrix/STATUS.md index 4664d251..67c05135 100644 --- a/opt_prime/examples/pd/http_matrix/STATUS.md +++ b/opt_prime/examples/pd/http_matrix/STATUS.md @@ -47,3 +47,7 @@ Live GPU faults#201: Qwen0.6/Qwen4 cancellation, producer-exit503/readiness, SHM KONI/Gemma GPU lifecycle completed; allfour pass cancellation/reuse, producer death/readiness503, slab unlink and producer replacement. Concurrent-decode isolation is being tested separately and not claimed here. Allfour live GPU isolation runs pass: active512-token decode survives other-request producer death with exact output; all baseline lifecycle gates retained. + +#263 held-out suite: Qwen3-0.6B/Qwen3-4B/KONI-8B/Gemma3-1B inputs frozen (36/model); scorer distinguishes parsed answer, JSON format and exact token parity. Tests9passed after9expected failures. Hardware results pending; #256/#261 remain open. + +#263 evaluator verification:10focused passed; full prior collection533passed/15skipped/253subtests. Frozen fixtures hashed before inference; GPU route queue started, hardware results pending. diff --git a/opt_prime/examples/pd/http_matrix/evidence263/fixture-manifest.json b/opt_prime/examples/pd/http_matrix/evidence263/fixture-manifest.json new file mode 100644 index 00000000..28505930 --- /dev/null +++ b/opt_prime/examples/pd/http_matrix/evidence263/fixture-manifest.json @@ -0,0 +1,74 @@ +{ + "quality263-gemma-fixtures.json": { + "sha256": "0c24b1e759148dd7de68e15fa435e03c77a85b947fd6e923eee46af8c87edf44", + "cases": 36, + "length_ranges": { + "0": [ + 38, + 68 + ], + "2048": [ + 2034, + 2048 + ], + "8192": [ + 8183, + 8187 + ] + } + }, + "quality263-koni-fixtures.json": { + "sha256": "22f6e5b117913530ef6d58ed5352bb47684d55409535157b75f0462d64797eaf", + "cases": 36, + "length_ranges": { + "0": [ + 150, + 182 + ], + "2048": [ + 2040, + 2045 + ], + "8192": [ + 8186, + 8191 + ] + } + }, + "quality263-qwen06b-fixtures.json": { + "sha256": "ed8adc8fe7bd7e6b5dff206ece812b74e398616a2b043092dcb3f5e26a20a27a", + "cases": 36, + "length_ranges": { + "0": [ + 40, + 77 + ], + "2048": [ + 2037, + 2046 + ], + "8192": [ + 8183, + 8192 + ] + } + }, + "quality263-qwen4b-fixtures.json": { + "sha256": "701be76546e6fa3c7410475bc026a50c21c677abb7725565fb650b514fc71c98", + "cases": 36, + "length_ranges": { + "0": [ + 40, + 77 + ], + "2048": [ + 2037, + 2046 + ], + "8192": [ + 8183, + 8192 + ] + } + } +} diff --git a/opt_prime/examples/pd/http_matrix/evidence263/quality263-full.log b/opt_prime/examples/pd/http_matrix/evidence263/quality263-full.log new file mode 100644 index 00000000..ce96f7de --- /dev/null +++ b/opt_prime/examples/pd/http_matrix/evidence263/quality263-full.log @@ -0,0 +1,46 @@ +............................................................................................s..... [ 17%] +.................................................................s.s.... [ 31%] +.............ss...........................s............................. [ 44%] +........................................................................ [ 57%] +.............s.s...........................................s............ [ 70%] +........................................................................ [ 84%] +....s.s.......................................................................... [ 98%] +.s.... [100%] +=============================== warnings summary =============================== +examples/pd/native_kv/test_native_pack.py::NativePackTest::test_exact_bytes_and_independent_lifetime +examples/pd/native_kv/test_native_pack.py::NativePackTest::test_exact_bytes_and_independent_lifetime +examples/pd/native_kv/test_native_pack.py::NativePackTest::test_exact_bytes_and_independent_lifetime + /home/gspark/workspace/aicomp-heldout-quality-263/opt_prime/examples/pd/native_kv/test_native_pack.py:24: DeprecationWarning: The argument 'device' of Tensor.is_pinned() is deprecated. Please do not pass this argument. (Triggered internally at /__w/pytorch/pytorch/aten/src/ATen/native/Memory.cpp:31.) + self.assertTrue(actual.is_pinned('cuda')) + +examples/pd/native_kv/test_packed_lane.py::PackedLaneTest::test_owned_packet_and_other_lane_isolation +examples/pd/native_kv/test_packed_lane.py::PackedLaneTest::test_owned_packet_and_other_lane_isolation +examples/pd/native_kv/test_packed_lane.py::PackedLaneTest::test_owned_packet_and_other_lane_isolation + /home/gspark/workspace/aicomp-heldout-quality-263/opt_prime/examples/pd/native_kv/gpu_batch_decoder.py:93: DeprecationWarning: The argument 'device' of Tensor.is_pinned() is deprecated. Please do not pass this argument. (Triggered internally at /__w/pytorch/pytorch/aten/src/ATen/native/Memory.cpp:31.) + or not host.is_pinned('cuda') or not host.is_contiguous()): + +examples/pd/native_kv/test_pinned_receiver.py::PinnedReceiveTest::test_eight_slot_tcp_frames_exact_at_capacity +examples/pd/native_kv/test_pinned_receiver.py::PinnedReceiveTest::test_eight_slot_tcp_frames_exact_at_capacity +examples/pd/native_kv/test_pinned_receiver.py::PinnedReceiveTest::test_eight_slot_tcp_frames_exact_at_capacity + /home/gspark/workspace/aicomp-heldout-quality-263/opt_prime/examples/pd/native_kv/test_pinned_receiver.py:75: DeprecationWarning: The argument 'device' of Tensor.is_pinned() is deprecated. Please do not pass this argument. (Triggered internally at /__w/pytorch/pytorch/aten/src/ATen/native/Memory.cpp:31.) + self.assertTrue(direct.data.host.is_pinned('cuda')) + +examples/pd/native_kv/test_pinned_receiver.py::PinnedReceiveTest::test_real_multipart_exact_and_owned +examples/pd/native_kv/test_pinned_receiver.py::PinnedReceiveTest::test_real_multipart_exact_and_owned +examples/pd/native_kv/test_pinned_receiver.py::PinnedReceiveTest::test_real_multipart_exact_and_owned + /home/gspark/workspace/aicomp-heldout-quality-263/opt_prime/examples/pd/native_kv/test_pinned_receiver.py:25: DeprecationWarning: The argument 'device' of Tensor.is_pinned() is deprecated. Please do not pass this argument. (Triggered internally at /__w/pytorch/pytorch/aten/src/ATen/native/Memory.cpp:31.) + self.assertTrue(direct.data.host.is_pinned('cuda')) + +tests/test_kvstore.py::test_second_engine_reuses_chunks_from_the_store[lmcache] + /home/gspark/venvs/pt213/lib/python3.10/site-packages/torch/nn/attention/flex_attention.py:2481: UserWarning: flex_attention called without torch.compile() - this will use an unfused implementation that materializes the full scores matrix instead of generating a fused kernel. + + SOLUTION: Use torch.compile(flex_attention)(...) + + If you want to debug your score_mod/mask_mod, you can set: + torch.nn.attention.flex_attention._FLEX_ATTENTION_DISABLE_COMPILE_DEBUG = True + + This will allow you to use print statements or breakpoints. Note: This doesn't work with the backwards pass and may produce incorrect results. + _warn_once( + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +533 passed, 15 skipped, 13 warnings, 253 subtests passed in 160.17s (0:02:40) diff --git a/opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-fixtures.json.gz b/opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-fixtures.json.gz new file mode 100644 index 0000000000000000000000000000000000000000..4cb6f6c0558df5072b3189bdd53caa0f93eaf2d8 GIT binary patch literal 23984 zcmeHvdtB1@{{Q!!v(0U-TsK{oa@L%7F)cGSZ~R(w&J{XWYNSXPZ#WgNC@S#VY*~_8 zTAGr0rllnYDJCiiGg4E$q@<{bq^PKfHv|O)epI_vJNs^K-*X;)f9L%lpU3AdpNEeZ z&-e57d_G?v0QSxQdb0kinc|7DTNb~1Jhv#{nTn=n6Y&85kN$pgW)0Ns4SSbf$$K6U z$Zc51&Dsg#r4KLEzqWQwg$v^3$+p68%=%E5KCrkSd4{%IsdL*z!}QEB8si9BKOVgA zo1`93?|#IlpbMaJ?nedFx7_LLoyj^PXN(Pkk~z~c*bk@>DH~!wJ{}0WmhF0NOrHR#%$QCSWubdPo3kqE$MzeM&Su@R zy?wl@gsMIjz?oL4+yIAHc`P(HXizl?z)S?yAOo-XnMUl|j=cH#b8QIKiM`Vtqtz}<-QsV^jOJoC+DNK!&_$WIIN0?4Zij_b7r8Sb(h(|(c+=zc?TBp zf_<YpnoS%*4n}|>4eh31>$yVC~ z^@rT~3Ki)dh{^@I&+;|vl4|A%balltP}8l;0&o~Jo=Zgnrw}TUFE;n{W=X9>D7?AJ zlYoZK5dAtknM^&Z5C{+hq%CmU@kIU^PDK70i&8r`QevjQ=j>E#MPX}r(wvv)>^sW` z?V+d}fF#_5E~_m~>e*ttoO(D93Z~C(cDWPTPpJv)bza(`s5J=z_V&l!)s*nN6%~+1 z)Nz^Xl`2U^OCFMU6wtLf9@~LIh4s*J+&OBeE;$H(I4U7`h83isK81=|< zAIV&;=wk~t^Ug8qDQHIlI#NAC5o%`R_~$S^*TK9kbQ;d2rlO8MZ+MG7Z?`L1AX=Vv zmfgrn@=Q5+tEI4J{tNr~KIkn|*CU>Ayt~ATrR(Pf@@j+*#YC>>Ko1-ddgd~Q;`k)H zInOKEnEUsS!j^7ZgV;N^-y5*nqm$utg)1{zp=EOoeb66o@Q>1L8`Cqbt z(hna?G*SXzwakGhy>)dNf9N|H&)Jyv+Tp>eb-N?qS~{Eaf9E)fZ^a)(-!ty3kC@yM zuC6}x-O}dhx89?^wyX#J>lkD+CG46tw${CPPpgo9UKmC@tQVW552^9(0p&EXh*!^6 z9?o(0X;39;GG zcYwYHcETqb*VG+l)IqQ<2#T$^ss{19Q91WfJ^KQ@<3M<+Byg>}ep*|_c_EM1QZd

KBCzhOj|OSXHPHsa070u)z_@MAse7+xkfp1QQ3$vFVMu+TVs$p?HBbG$i5~mC+jD{tRb_ba~f0qB&b5) zBR*|WM}y;E(VSSi5oa8~QO!VYG*9OP3 z6WtWw%h>B-{nqBiS?w$uh^%fiU#b8Oz0OziH3W@1M>Ao;n8Zj}G7f^2j$*H(Px+wg z<7Pht(iA)HX<*^=vrV?1St*+SQUUxSW?I?hv%y!T0-_Jy+q!FU<}`e=7(7K%D?&W< zkU3QivHHAVt%rqW)9-adWu^>;azf*v0cr$kBy|Np zTAG7e)YhkhH8?eC5iMQ(#2W)uAY}!z$(hFG2F&20HT*fGbY4t=sCZ@a=voEU@(XVa zze4ly-9=9iZ+YP{ukMPLG1p5mSl8;Clg_*#=c-`QXiS}V463hcbm2A%zlfb6MBx#x z^Nrq^x!YFlA`Jia$La^9*^~7%6kfARtOz|ea30a5nqZ>!4N30$oe9)qtHrWND=5hw z0e9C7UsLz(FDp6J5bk(E9MWsyti0rGLGN_q=QZ4aY0=N`A)CCsWUK3>|0gqP;^4_4 zPYxJ(tt9l`;$MZZrLQilaM$C@!asrR&C?P3%9qeX{_Sgbgao1EzGv3#;%*v+;FW)nC%vOs*2a+Zwyr$yF%U= zD1JwmMUI%36u&7n3+_4jLEAl0fC1hNcy$pQsnfRg5NXqAd7P8Q+rs1DE0&n*D6`u@ zM~?W}xDTaSNTwbm^}U}S^LlyAu}~VtKAc^6jdwm!Y)K*%i*sO5^OhvAjhH54`49HM zy1iWeVo}{y0d8H$r~!9QyHh=9;Wm33V7 z#`P~QrAZ??X22zkBc(!Ox`64~qOeEdR6#e>DYa-~)*HCds_xWFi~q2bOtYOWQ;l5zqn;6sz)DGXV#t<(iys_fM{LoTc=D5g8ObT97`}U(RNou`wjz!WWrzv&ydpK?h*=LVr8OQD}q-|$1=gqEZXxr~0R{@Eetk)Pmt0?7* z6}9b$ipubL!W${st>k2(pKsl=g2}=iU4tXf4Ng~suu%xpJSoI$AB_5GX%*BM3Hi|T zN}ME#eys$|YXtH1Sn`T|j@)t=*$Ujs9j$%`ol z!QOUvrO#6sVJQQqXNHlnB@rSS8aTL_W2(BtAdRVeIh&O(IQnK~gL+oNPeQ911%5`= z&&o>A=EyIW6&{#~P_r=#9^a{MeZFQMD`cp{7T_IDScY<$1hm&QWKp2E6V+KAf*JbK z+~d*O0+pna4|lTHH1CbJcnNQ0tJ7Wwe&C={GWKKUt(L!iEC-^E95wPRNf4uUNfhK> zpke8E%E=f4r*S_yF;&vXd!WC}SA%ATFw+wjVl&q@3Z9a?BBdP7ZvDHfER)^BNlrZ);e| zlHRT9Qqgs6x`OjYT4c4gMs&sppKRI!6{b+gOI>aIO6``%Pk8Eq5GsG)+wK`3dDbDF z+SpSOL|oYg#=VJQ6WYFJz@taVjp*H7tG>amWQjx(-5Ry-XI`Gi)oTI%WnFf@Zcm7* zZ`Ku^0&>~B-2~yqpv?c&5>|V~tZ%--e-s>#Rh~ zj&o;mEmjU-^z0qK)-N8KRV?IvdmU*Qm^%E1I7U5kf?rVs3uzIJZKoXw#2Y%?7EF$E z0DglZ(Y@Ew77axro;i=AD)^jvht~t(VsuY(KJK*ICwrQsM zm}NQ}bg;LA+p$|+705l*H!+GrO*4TUBPQ}WTycm4`e6oP+AC^jnnW>=FHqWFHkw`+ z&K;a-N)#xr17vY;@Bs3J)0$v*+-XKHz&9vIC#*`OgNtPJ>^k-GsYjRo;-P1~Wi{{^ zu=g5BITE{|@8NY=^y|o%;ihQn?3{8Q4YnTDW8@d91=!q^JWxrk3HgfW) z-Tnc;lKLTN-erOgw9n5J?l1&4a66RwDB!!Wr@gSy37#6!{`Jp#;bbH~gZy@$bGF~! z%T^@jVUK}-Jl_l3(4Sx$!L%NVXp0Qc21WSud`n2-Y{^u*fMjRg31*BgY_)I{f(=G|sZiV%-2D%O>BjF_)bwDcA+!WuIY)65$?@Yjh{?R=;5bN6N}WHW0i*ek#~*4q4c4M zySyXBoQ^mvG7F!zYD??X>3TE;%z5u3&eFlG@X+IlvXWfGcrWgu***7YLVC&Z}w zClW~$CvvUc%W){xf$M}8Ja2nE>RU!z=NaM{%bY=0Y6s zJ&RAu-7c@rc2rxIi@#o$ha$CQc}rWCHc5Vx^mc7o!YL~*%Lz0;c{9uk6cvyNf#~x2 zYm44VCrOxGaMHw>fu|ul=FSeR2^DQg>?v{iwsg<|crtV9gl+6bo2L@YVBUys$Ruq? zw8aMVl`JBOsw}Cxja+jow<)4X87{l_sT-T1}l{qY(IDLh4z=b z7JGGj>gLL)rOX;DzO|i%E17*G7m1jyOef?(G4V)3+sSC_L9QVM=FvX=G1x@Pq9%bRIj;_LOsK+T2#&rRy$3ksPYdiy3hxEf&B}-UJIFyuy6DLPOPX8 z93aJFkL`JJmBUf3WiLXW_VW_~w@24Hi0uS6P&r>VX4n2E7Ur=(4^;E(By6uhOTzeC z5;mh#H@3O}rZ>36++4S^2BDf5*KK66lFBeQCw}Q$-y(RD(7dWd{un2tjMx!4J!rpX7b>j4bV*(HE>);RzYw`Gxyj zC2XvlwktOzmsTpYva7xB^hpS~Lawg!V+$rS*r!(Gq%HU{;3e$&#;&yC@8wG>-!U}>P0pS^3kGla$~Xht zvNvNToE(c4@da-;W?zYoZgT-(tCtnP(EYnO-=C1iu7(%3Ju@GKKrP`Kyq9tDY}!GW z+_R=E-7)p4f#*QQD0O1Fxt<;n&3<+~ip_s^JT~S&cRYeKjh(;nXF0g(Nh)!sNU(Wn zrH`H~c^mF}nGjanDhW&&pESWWL3$7n+$eJqbgDlbE_Q0dM0q0%5150Jt9`BVZY5*@B#rv^F28W9SUwIFZL0z|60{Sdtwh$%!=Di)Kl8* zLr2KZc6>S5nHv}|E$&$%o5Jnn%_K&AkXGzU3?&gnLrVKom+R+Z(ed_J3!ZOLDKle@ zS3svf$Gx#zv9RrEH%`bdD=nRhsCn?Tha@t-E^a4SE`Bt$q6R~e0Fg2@$(c``4V_Jd z$~B*47Q?P=I7w`7N$vo9Aq__)PppmOSB8wQvtO&56YNiOSU%^!4o%++|LF9!prL~} z8|S5queB*=_ZAKnTdsD`1~`=IlQ&pl&FTWT!TeHk3#$*<=L7fmQbNP?^5$!iZ^JL< z2I;tqsuSeWlpIR`7V98E9WN56H#+5J^<`=?xRroBE(BME=D_eKl7PHNk@*3r&ap5? z=;ZlsFW4n-SjAxlvAxy=Vm==QZhIaSe>xm_zwvO)`J;y;Oiz0_HvKO@93yHsg8JTnhZY5?YlF)yaXEZg75N z{O>$ZlNxihG>L|NMw3=OPm?M?HF49osBpR^G2YH5PL(=AVjzyt!}hTdyt_R|Tmh~O zE6}kqM+P&v7czNg~Qm-8|xM7I)O#$_t z%_-?mVWm|y=<#~n*5!{!>YsV@k<5d8z6X|sfv3`xi@#QyS{!~RO@&20Cr#yAP?P)u zT`S5&rD9gCPhdb#yC5)E7MZyL#nzJ$JM>XNN8NtA3|8$-)^#|cY$ix+&qEi0$k=Qm zPpVWjCvYjuy7K%28KuGpc8!x`Lm%L$OD5fhdc-O;&4%c%IEhJaRh5%$WEs{d8yQwL zq&@Qrgc>2n_eYU6&iVI=tStHqA}jenBI|!6vZ4)=HUb8Ch0Udko!896hZQ@6|J3$z zHW>>${-N{LFDvFBM`wHb>BwoR*yM7r0Nb}FtPa4J?&u1Hk}ur*XWeN4xsry z3hP@0_s6>Sv}2QUWwAUxNw~rM^F$9$ATu|!WZ_RsAQTcEG+Smyue1H{0z` z(svVOJ#mFtAi4XL4Rt+gHfm357sU^gd%t1yqr5A~>RWH?Wtybx%E~5BM0<_GB7%=! ziH!(7UQ)lEWd&bz2&5N!`)df+VUH zo>42^T}3#_*dW%?zssB!pc9qB@BSY2_zm|eB^OlV)fxRnQ5<#rI%dl*4)-Px(L_i= zZE#@^kU@`bhq?{Jjc-6h2Z?VkvIz>&mlZ7V4wSZZHK}Ho{gn{MzsRFk+WKbsGME3 z^s*dhy%JB}h&EgEyk+<^gJ&bZG|MEv&Zwn#+Ko89A)NPveT-m-D^Ek=Q-Cl<_>{hq z*@2PBWs z0I@1-a+>U6_7cykzNiF1AFZKgD8=HU120RXf9T4TyI!Jts!#5&nTW$E7u;ghEDO9w zLFr^TD$Uf>5{7z8E5Z$;?@r6zoiZ58VQ-87Ry})JPx~EL`02-HSz9`^`QG*dht-%w z!M`%Ufv=P1j}Lc*-tBFh{un^>y->Jtn*%k8jI6mB7+@4$`t>*0&kK?r33k2LJdp-l z^1K2Jx8QnU5iOQ&IOZ2wuls1DY+F0-fg;ymtJt2&`oX! zq@aO{O~1VRsUD1Rf$N0LiOt!^{@|fE+jmjEHD6uYpX3qqg3|v}1K9ke0qhffbwNBm z5|I^vcfS z*k2l6`H8@}XvN0yU|C_56)p-3>)=&GRxcK%h$p}p`&w$c!Ue}h1UGdjR68b$>^mC@ z`a;Tayz@@+HSak;Kg06+D%bT*+7Xu6JNpUqR<>5RTkLGDZ?hj?4#=}fxnZ)Z%B!B} zjQ<8VY+@U5#5iXa(myzeRTcb#-aYC)q(d9n_}=49TnA>p34WtY9R7{HZ<|H^Qqp zUO7<#sxSw;WhCa`aQ52=sk>?e*;l|}S-O%(X)}7Do*-4schAd9{7Nd%RkP$3MR3cD zktJ+6O|^fkDrNJJ;RIe~(|;S>S|JJZ{U0M?9eZUz3}O{(=hP~F;YA3pA<0PQu@02N zP)O5$$r67VSAP4IDehYPXMQ@LWji>VxS0`WMdM@yao<0#Tar;^ogh@j&3qz~yP#}7 z0gxI6Fcc=TSDQ~%1Y75y-XpHrOd$-R7w$MZir_AB91l%!bpIN0^>JNF5FBM%yR_G5 z7ebrY)UnVn+J+KIb6kCW&98%VQ*fmx-iHk@o#-fCV;$s)e}w z&LjW&o*lh+ACM!CJedVZ9f^A1UGR_G?zSuC*>Al{hy043m9x2e()hjL<4v%5-|EvI zuYb_29k5%|aS;k8e!X_fw6BS4R0Ds@;h6R9kS}sl$euyBgj-kiMX%5%EnC`nVb@O4 zv)#0EefD!NZ6*w>|d=1wSiKP4mv(+Ugu9GJ|Mm}>3HCne& z^29BS9N+ddQZ)d5Zu0Uacjp;u)%`MRp*mYx&UNECk^3B5!PKjbm)za&%aarul(#i# zP{?+J^u%~lw1ZZi9c=6NEW30O0Q80cxNO1+N2xdNrFz`3ucsK@VhkvVj!~R0?6de;VI^DbN3V zF37~^UT3t_BWM zPsT>>u5hrAA8hDmk$_2eqpA#k9^S)B=*O(X21g|g6<38=>o6+uPz8~YdMY;-nb$5F zwcM@`*%TC;9f)L{qdHAzI$se!k1aohwkDp*QXX1a%DC4ky!R*Z-bl&x`%%3A0J99m zEHM`c@nK%|vhS*EJ7$`sNW?=ih%1q4^E>%7mN#*}zfSlfanb6ZX#E@MnI< zEn1dkxEkmb-lUod*aBawOZ&6tdNq`MD(upT?Yp~^IHXS}Yxf=u9En?1{*MN;@!fxg z)lY@ozX-Xq4>IZHb>xa8;LWezO$J}o=bpH;`}Cc1)b6F-DGlShmvoT*nv;YCC%X!e z?z-{E&hCKgEve^hhR>JQ^h`DSG&k_I>VEs0%nUy#6W8iFML>3Tj_&)U)cBj}Mitf| z)4d`SgLcu@j}IM$?Cb1*cqW*&V}rQz(O74&A!rxY!RRw?FsrK79FThPlQtTRbnR}- zD2Z3D(^GWOa;C{YHBCa`_?VC7( zJV!EN<&A%=eQgLU)q$=l+$BiF=79#T+LgWw_c}bS9v<)@w=(y+yyd(;sAYaNpdnln zYDo#nhQ5g}@uGmC`2q;^Pt@eN&NVXiR1Xc#ZX0UJvLo3dEWb zKJuPQuL{P|ru0Dtlu^r762zb$l~KFxqx^l}f*)b&EJC=pCtY^dkiDs`)G!7KK(ayE z3>Z?GS?a**KL4<*bxtk4VHqU1R2LiVB+hiT({zeBydNzV)vJeKK!CCVEtkUq1sZkV z=sk9ug3XQu&WGCo=@f9mYHYH9;-i9r8A@}f!v9j44YvDXn9+wRBI%q=?C)?qEbFAB ztU1Z?0xzT&_@PqrA@2_AAv~P7=?|*&FKwq&dLd-8M<|!r6 zGjVv$0#?FzLMyuyI~dD$G|6y@z6D=7sY^v-hZ&jDgpk#+%4(7t2zzTwHT# zi$h9EmUPHem2}vstg(ovL`4k%Kbvm#3q=N1aTCjNHHJ2XA-^bw{eBVrLkh=LGrMqo zbM2Gq&$e*d72oAHFRXa#@AC1{)0t;Blc0exr2O6a(sL=3jX}cp&(;GOa3j zs)2WblrTg9?G`B|2pLQWMjf&0jzREz!G^peUfuy07zvxaMO621i42s}NGwFi7FH~+ z;ob=APtpM$dE_|+_}}ki>hikpCsS+QM~8FE@aattYu9*QZaL`lP4lj9X^b$T0&JK! zZv6fkyPX1GoOicBy7Oasp&P3iEZRXobBFp5xC%M#hHuh=7oxD%*2T_7m`Z1#Mu6+1 z(7fh8!M8i^<2GhxNQbiv19MZpOkbC3(tde#b0t+pZXPH_vAd7{0FF@6KLJN1xs_Z? zP$0`~%leC<{;b+v3dE)C?jhRSkl=nKbQcy;a~Q|8{xY@Vss+w!S4(Mu_B|??PR&%e zxg?xBBjN?_GeY%e5aeykfmwG?$`pS;O>xA#oo}-K2^_o7qd$RTKf*5``r>KI)uT)SQjat}1f!NKsCP>$-3n zI`_ut=)nhstwyCqDPg|)d3 zHgk^4%p1Po0e|n|SXr2bgYMj2fAA6KfvVJ;jmV@$Ew;qz?ZZ{CosYly_FQ1Y$e=~L z;XamwZ}jEQts;xf__KSaWF02a1R2Qt2qexO7E`gcAZeycF)GS{)@d`aB|!q7*h9Z1 z1aYyoHWYd6ItqLR)_J4tWcd{JROe`H)!`foy*)2!@EbdpB&u&8Y_RHmk&uM|=X`rB z?@2;Jj#USd5V>(&?JT6YiYrie{s1f+w7{Zm8NX_)L(}Yu8OoDXTsc&ha0e6L-_e9F zhsuYj8aWiGSOj$E9CEkO{@|Kay;R=v(qN!g>ROX2_P|6j_$Dr*Ea!?rGp~abG__sCLjBFeH&bI!#d*=QE&)&WbX6+PkgEZ5c1E@ScQy%5=YBC8&@iizcTxzyL>@< zhE`UvbDW0tEIT__e2%9R@7t?RHgjPhlbnD}`8Re=2A|^Tb^ayKB;W67mJambzY-n) z9p6O+)M}y1Z(rxViqxu|>>a7v|4!B%?{aMAjtA zw`-^IUE}`FPV(s+QWgimNQka7aO-HKa*NC_cTfF2cb9}oY#T#yM|Z# z+dlaP2>7EPdl$9o*M$kWJ9=>u>ahRQc8q39Uchiw%NDT9`P%Wi;blq&A1xh}STNf8 z$T>DPX9gur;n5gLk?PSSaWB7q?04vBFW)L-o^u9{#j6Pg1wiyVO@crUr@P@bQ*&}Q z5VAZ&FzwJRneE3YI1wIs8=_y`-D z`yFApZuZ=J$2s4oexYrxch}rVsHhGprg){8*tx#>ZF`F?XUub7F4W3{ev=&))l-{l z@&1(gxDN(T%@WSl^YZp`*49#vhO+FX + + def test_http_runner_supplies_required_request_id_and_frozen_tokens(tmp_path, monkeypatch): + import json + import sys + import threading + from http.server import BaseHTTPRequestHandler, HTTPServer + seen = [] + class Handler(BaseHTTPRequestHandler): + def do_POST(self): + body = json.loads(self.rfile.read(int(self.headers['Content-Length']))) + seen.append((self.headers.get('x-request-id'), body)) + self.send_response(200) + self.end_headers() + self.wfile.write(json.dumps({'choices': [{'text': '{"answer":"73"}', + 'token_ids': [9], 'finish_reason': 'stop'}]}).encode()) + def log_message(self, *args): + pass + server = HTTPServer(('127.0.0.1', 0), Handler) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + fixture = tmp_path / 'input.json' + output = tmp_path / 'output.jsonl' + fixture.write_text(json.dumps({'model':'model', 'max_tokens':128, + 'cases':[{'case':'a','ids':[4,5], 'input_sha256':'hash','expected':'73'}]})) + monkeypatch.setattr(sys, 'argv', ['heldout_quality', 'run', '--fixtures',str(fixture), + '--endpoint',f'http://127.0.0.1:{server.server_port}', '--route','test', '--output',str(output)]) + try: + module().main() + finally: + server.shutdown() + thread.join() + server.server_close() +> assert seen[0][0], 'stage HTTP API requires a nonempty request ID' +E AssertionError: stage HTTP API requires a nonempty request ID +E assert None + +opt_prime/tests/test_heldout_quality.py:87: AssertionError +----------------------------- Captured stdout call ----------------------------- +a True True +=========================== short test summary info ============================ +FAILED opt_prime/tests/test_heldout_quality.py::test_http_runner_supplies_required_request_id_and_frozen_tokens +1 failed, 9 passed in 0.97s diff --git a/opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-fixtures.json.gz b/opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-fixtures.json.gz new file mode 100644 index 0000000000000000000000000000000000000000..2b2316a7a031cc9ead0dba182ef9f7d0b31db731 GIT binary patch literal 31333 zcmeIZcU+VA`u~4gwP;nSqN1{D6%ZK_5W=R4f`EXs!xR*vObL6}1x1#!f`m~)KoSU& zumhyZ5&}dB0YcaikU-dk5t4j^c4&|7=N!N7?|eV~o%@eG?uUvu+w3OkZ{et& z|Jb@dXU_}_L<#iN%&{4p2QLgYcG}{3WJQM`hP2~Vw7*c{LdP|IPo)x5>Ctf5>`~Cm z!_oUjzVo{H`R$Y+zC9NoB>*|55HY+vLE~iVCJ&a{Q5OtV6^XkKw2B>~i<xOY`_M(91C!9!oQE-#dP(g85b!WCs_yz?VAr*Rh+IA|sGkh%y8HmZYqC=C~ zSJ`;~9}@%Nq16KIP_zQ;ap^)Z>AcjTa|d97$IGJ33L-5f6-i|~z0{))4uX##N(k{z z6$X4E++c}7&y!C;(h{Vs08og?qXjbsk@y%3`x;)F|6}Ww0Vf1twmPY%T)d?&3!Qqy z93dc9Al+hn@2hOpDutkciM07s$EzGleD;e2zTn@iC-H-frXU$_o^dPf$}!Cc?%brpoD=-tyN~uoci>FTOw$v%%9LBWcZ7N3l$7L( z6Rr?-65U8YnD{X6QMYRc057O@l=z~is0BCY&Ii+<^y%rJp)_i*)E~^E9L@I5c+(pt zekXV%+Ryl*m8A&2z%;k&;5b1Oxu7$cm^(^7Wp@W%t;I3SHLx;_3cO4 zeGhx0s_hD@$5DmI%1E$*0+Y_b=Zc{~7S!1mdoc&)DC=hCs< z%VX0`F$4B4!M)goXZPnOq8#)3zlmX2`3{1{W;&&29Bz(f(M)!_=VSVv3X_lJ_WN8& z>JqQjVI%vKeu8DpOg&f=q$rT)_~`07-=a zsd4Lb^Q)+`IZ7<>ZA%C#nzd^fRC0}AM$~!vZhSTeuFhNtZDbtTx~W;71Iw5jvY(0$ zh2C)m?2ubGn-p`e3`EQr``q>qAesS-6>rJvXM~Y#tklY!Z&xF+3+;gt+XCd70DpO+ zbg9;Q7Vf1FuKJ`~2Ea$9Zm!o5 zxWd#K_`FqN)>PEQ@{1vZxiHVbxh2BT=7UU?MO$2u)LdzG&!R135!*xcY;NHquC3Mv z2j#OTzF%1pK6n3N4K%JWg(9aC9Z{eRBJ4fS2hUqOzb!D>Fn@48k=|^B+*M@#J@n08 z!s5Qfm;Qp6s?EcQ`&K%%*$yE>*+#^+lXG_#<_;&oDGT4?ka^Z3h#Q#!smqm6eXQmQEC6??1wh^~_nZ*|St&6Dcme zxVb(zGrBv7*%oS_Ib$ATKiA3H9fXgP1GHBwE{(Qs-3-kFW|}ba))Tvf1Svdz0Sy~P zp$(P?4nmyy;70Wi3wIiK9|hbPYWA3^>sqC# zy9BJ7tcDhZ${v|MzHSr@6kc;@i3IL)^KlO;KkNa$K;3i@DtM3{S6MRP&=C`M##hK& zWpM6Ax%({S2FB4_%P&-k6~d2g^jq41vD{fBZ03?iC=03=1}p-sxrc7fNS+Vn+^!B( zPhbKDuLj(u>`ew*3|?J#uRV8 z=1a2@*wpgeb(a6Zg~8S!xw*D#;1cI%shojWcaGll3h%@v2fX0C6cE$MOR0EdS8mgY9&gxPC_ zK-Od&c7kuoi7mL95CUK{RF6aj*y8=w##W+xe(}ZiM*rF zcx#HJYm1g@f`du5(70d#o}WSD$DnL%6Or``Y!XZi?M=$-l!g`-dxETA*-V=r9_o)e zx_V{FsJa}!cd!-LKdl!A@W7JhD0Swc01wwC=ED4ZU#p8;QCqdb66c2VdV>OJa5-hQ z>x1WQi*2*a3l@%%EeE6$VxqblPYc@-CkG(=j3sv;be*j-gSCUqHzoJ%X|xt9CCnNUNx!R;${_lRIMg&e*>8KRdRArn24^ zoLWJFbJbq4H3dM3B3A|u00xHF>;0Ai9n&}3MOrcI)uvka3-n}iPDP46AjrVuBGPA3bA# zo#%MaVN%^IAuW3K=JBA$ZSjEF@AjJEq6p1HW9sUelna7)P<(MuJyCvxud?NZnk|?j z@;+dq`Bfj#j}bh=>cMOCBX$4I4V$ol(7C9%@o#?F&(FY~4VCw8xF8`$%g?itksWGo zz6+bDR7sp|DCCF|_Vu#rRl5xDiG|D|mCUAsp|m|f--yBmd*8yxQo+Tzc1e>~ zJ*WdY5NJpBG#q%G@&E;w8kDk}gFb;V-egl-{C%@cce~w5P2Ul=e3D%!R_IMb&Jch! zcR!O^+bibLkf~sWQl_`NLydJYM~w2%BR!pm(QJRS4=I7M@iLiJz^#XiIkBxg#bwT^G2xm{yL8BzjwwMlHFdjo$+sPy@mb0R~*w=xa}{-H=5v2-vHiPJ3>-YL3@CW*!wn8^`$AKTAbQw^^E@ zZ7*w`in^K&6=`(9++aFBFu9Pjgc>I@Bym0o^7-)e7}RwTuFpvV5tEL1@VLghCFz8I z*Jya;=U1m4J&U2urbBLJ8>8)3q#)RJ;`45hehI=fme-8W5peE%)E8v(j93daTNd@L zm6h0|MTh4v4AW{Ji6j|xUp5yc9us>~qT7vXdh<@=s(|(hrBz$PCBCjW5j3ptbJ{VW z%s@xstAYB`BiRK(er2bq?TbH}X;(@YK&2*6NK_%xqu~ZwuDCC9;AAlqF^gEK*FG{e zs{b%hut}GvrGJlCHwns*%%y&|jTqSjiMgU|=snYLfcWP2g8P>cJ&C4^7Klqe60hQPooSdsps`Oroy!i{l$D2bqBXYO-$mZijEpS9Dg1=sXl+ibK57> zQ(Xjm|I#&NI7>ucDP8N~U45g|350uVkqc%kW~4l4?}t#$WcWVotUB2?h}w&Pr&50| zxsfo5H8TTm2gI%@>WG+WQ|;6v>A{w7?eJ~c==78?U*$MYWL@?#Hh?7-JYH@I@I@7h z_B~FykuQ5aAQ_z|nwzTA9PVsz!cM=$5uhOxlLosH$XyIDv{>4b;P~6!9tPq3n7Zz@ z)0pnobHF%dzq6$`N0_RmyD3n&KEp!!Oh6K9H$Rl*OX32?wfh65B{wSypl;wLrke zRZk@8m`v6Rn!v6Is~vN~)RrEhyQCoEK2kNDN~(>q=Mh#_WKPYo7jTT8RDwqE!aN`o z7#22P97r3%tE)2yYRof(>GKKrE|<7aReu&KFHE4EG2N7xsXE-t`;kVy)x5cC)^3kY z6>Ul2+Ra^KQyv;FLafcfsk<7P%t-=+hrN8kYh&0WZxyRpv${Uz5i0*hwP|>Q+fV^x z45Lr3UZBHzU@M!<&b@71h>=^H)n~LR?p~*Zy2f9)x(3O(SMrx3SWaFTHFa@w*z|O6 zKS`iGM5UmcNnDC=B0AU|TX(nF3C~67pp%IRzQmZ8te%s*eGx-0GZxlvG&1N@w#FzxrkWm!-UJ0B2oRt%RV$T^=o zj&8Ncs7cCaw?5oWP2)~HGpJ5>iS7)@x(3L7)@c21HC6I|Y3T^JI;*ZWel&AjAGFux zbXf~@SaPm?L5U(w=#tC+I<2=^B7oQ0=!#k{)_L>W_%XE%#*g5W-2e6jHWo#SSW)~b`g^?q`68?fU$g+^0e@JU(% z#+ui*@+&|GS03db$O9#D*7#2wqeEZT7J^R&ux=d{4p{~$3_C{a5)VU9GM<$x3z1_M~ zI8?~#%FAl~hEiJpczf&E8um)wdi8-ZlZB;)Ax9JUeZgNIlb(FhhqRJ*qrl|-w6fN{CB4m)TwdoqV5e3TZ7oH=kT}=vljU61 zduDnAx)hcVH|ef#fnNjQ0- z*Rr;G=_T%0=Avv$Hm*LcA#3Gkxw=*@jG?@-aC^QnEmRHOe{cton|+`gX`|LZj^T&%NyUr0K{Rz<7&(%adN$q`Qj&>yB z@Y2V{4yR`9oWY^nRar|xle~{tU zp|D%8v42=uX*A!{K?g!k^OG&2t!Be%a4f0cJp^->51El$w}-drOD@BaC{rWzjV~-u3IMNEbOR z(Auo4X;(Vsb|I&%KwD*H=KW7)W&Ojn!~10w@{Df8t4}=s1)R4CaqM&5$;jB6l&qs_ zrw8>R+wYV?9^NZ$F)6ZIGX8TYlpDl$`XdPIQ>PpG+R zUyFm}rjq2SoINmKP9|q-50nHe9lInCj^AtA6C$6JT(f!L4CGMO6zOHHs(jm+2G&oV z48kfMai8I~8mpVJl$(BO1TfBvcOlA_Mdfg(_vp#>fG`b^*+Ye(dtEv0WsU3Okm8#e z2{Fo`Lc)yXTDJDOu=>p$7eB|$(2~LAGOSF=(ir4|olHy4Qe+RE@Inpq$QtwOF4#wB zsQfJ?Zo1~^ZEM}FOfENl~B(0H1aa?!}#&X;kIE=ZroWLaz^v_%3+X|iefluR> zDsu&3FgF0zuvcZ$z-@J$?Kot963q?{WRGOl-Y^i}q%E`f@>bO~m~BOk-<7xALeXG8 zVZr4$VZg!Ar9gBB)~@Qw`c-*>*i7ievwk;jr?^zrlo&Taqm*BrFLB7#y}U_`;_MYN zkf;J1EAxmTR$&8e7b`4hBu)$!?mkQOmcQAJ3S7B^x&m50T2r8wjGY_Yz#YhWaC*`) z60r(Oo)mR4GL?`Bi>o1`-jH%7B|;IWz*~-q4sU6>eb#VKAn4MJBZTM&E3B^IX;sm$OeytAfC;o34JsqjGsTJ<^c3Q}U$5vSDr<*c(K`@nvi0&SicYVgAIGfJ}3^^)4v_aENHRuthC zzOs4N_@n-r(;1(O0F9N+u32Uf?wCv0>v_MpsT3-)gLWJ$_bJ*~m{oqqV>(hdLnF0O zTbLOY<98%GLv6iO!D}di$N5NCSSdox1n|*x z84!2tnuTyus75+D6rU%7OV^(YNA=Fl%&Z&yG zN6L$H*F&>c)<}QIcJ#K;5r^)GbD?G2|8P@mdBxcj&vQ0KF?ep2J{lPYWJBS)j`{@! z%5+p3&@$|Cs_DLo>Vof9)N#~->xfg7;cUt5Zq#63%h0nqYtySViV^a}&8#!POI_Qn z04^~#myjFoXd<8AV9Ox+00KT;AlgSf)wBx{h2lF_e=-i)7Mhi+bn1B2SZUJ1epA!3 zy`iHsF_QV-sHaubcMc}1R$a?n=T50{#8C6+3&Z0?Bb(1&h>*5gJ|w1KTE3 z-AeYGTuXyMZ^FM6W?1WseDm`V-3z$5ayemAb^geX?VrYXAb%Dyb-JQsU|@Y>k{b?8 zFwQk;YV09wF;k~4W*U6b#9wKN!%Pn=E8*jkj5*A-ElEzsd}~(rJ1PTjchza*ng8e1sr~P%Qwfebbt^;lanxz@C#%!jzpqXm|21_=e@~s7 z|BgDf{J1(D+rd$%zr+LJ9oxnR4vZ(8aX)%)_Lyq+>zL#FaLnD03pw-~acc5boK}3A zcKl~?3OeVVcGTlwyuj^mng~Y@#t6_&Vot|1M?X=Vavor2aMJ;i9|^xLO6->(T1z?P z;rK%lc%*}5aA{-ANM8HOvnp}1i@M#H0x8kTBr{{`rAzrHM^LIbkikylw#S@7#duVd z+r=4FceL8iol&#uh34hHr6^#P4+fPC8M*sZi58&kn;?v2`UAJ_3zX{9&NrKkr%cQY zjue$Hs`p(`5ypROQL@|k_qplBTW(rimdX7aZu+BMKmU^%K#l!wP2Yp8`FM zNC@RwwB&eqx6ae*!>;qZV2|rMISKLvQ*EDDIwuR`FU-nR>FzH`SKc%*`;JSjD4F`@ zPU&VLe`CQ*px8p=RJeB$vh8qJV2@;7eB z*$>?gjfdm@aa(T3gHLih*pm}3yroIFmcvRr&a2HzcV|av`PBMn?fIxUmA@<&Xt#GD zz;@ePZ2BDmvS#*WpElCz(=d_8j-l?QMAfj&e}eEhSrTFs`q+9d75QXL|Mn^saP^#TmGF6IQ73eOIzHs&neub%v@(m`w34tK!LA@n&BnHV2*ASP|-nS*way-oIMhayMswC)J<&Gzm zWtt)f&m+l$l65imAtoD4S9gY5CQ{N*A&0-k0qgaAQpK%|13m58;)IQl%ge`AuJ7?#wHF5&Ae z18JP;_k}(mp#mQ?;sdg0e8JpbQ)rb*)B~>UuxLPPnEc`lC2ohtsn0w~n z&m>aV4H$tMND+s)f<7|JO|*F-};M&*FW?ZTQ7hG#~g&n z?nrWl#WMk`$tY7jtHecH7er~C=Mg>WN*l>e>uKr4w8HfFi{=+Ezxb_}uUyIu{?N-m z@`0C+FSlc_vRAU%^lbmy>38Phbc*rsnTt#I8cLG-WFNZKX_->2;UB^fd&J1Kpw%WnSXsar4Oj%6UT5U+ zr}CK9jjRPIk!e@yCpS1VuJ{jUocK6r#-(#++#b%1%K_n>yLY!-blm5+g6RsV3Uk(BZ{6FTdwqQJHFPzushf=}L^y;kn$e%S(O>O^;Syb>)~5 zpuzMpuNIr!{6LF)ehy+qeK3)1)c|Ew*yXfeySZDLy$@ z`kVAjTsOIs4h;7VwMO>7wfAz<8JPUl@uhIdte(h7!6#V0&$_S*z=@?@&sS zeW?1!`6fdlTrGHhcsUQtZUW5NxfrnL8|tvZgPY73P4vPlMBB{<$9bKfh zu`s+2&SZJ^&lU-1hAw`0&G2_Ca+E9fr_k%7SacqiSrD+c6qLuq3Z`co&S`8C_}ZrK zV$%oNqsxPndDwO0;rWjR&h>z2gbTv!eomavTuI=BaAgd$I8nfmbc+EeZLg0+}pw(7x+T=Q01>C;wNC! z8;Ja$K@A92G(8&di-~BX&wu4z;qL!n8eWtBX&S29zdsENzBqOG15P8fV3TAzYd9 z?&|Pv#B5AmK>tf_ixIcewSI?w2+-C1y;Casm1w1(j;Wx{YkSg|a#m8lXqPrj93M;z z_B1gt$Ci5%2jXY9_W9SkW*ypn?16sGKJo>LZqCt_G*_#8N|_B`&Y|~3HK(39bT`hS zkO*3H+ppB#oLsmy`d|4=!1gz56+9n|^sWAY^-}CRK8W&N5wd~RsCpUrC+)M5=~IqO zs2!JemD9%dQbYk3kBI6(tFX2$f^4Ljvk6@(1eT7(rpyK_xw*81O&XjnKSnt zr|)1ZC~0Yrdp}GBWmm(pGHzI^{7iY@O?^oBC7WZvcrM?(%EP~9yrzq-@8^MTo~r%8 zarqEMk>pejH5QCzp~1s(Tqo@MKjSY*=F20<%5eo;iTK(uV&8tTpvf3j%dtn%L^~s6 zKeLb>X3!m<{Hf1k%Ll{i!=)_&0$T3RVp{#g z4PBvW>57W#&M#goo2Prr_J41xY!WDT)WA&Wd|t^L=)qi9bg!PfV=gBTQeAK!Skm9{ zMZouW{HUXQOLcT9;A8}E5Nncp+VnOf{a*cfCMQ3swtlOO`hJ?qYsgn=0yUp+-13ZK zO$h+qi@#%6(%qQmaH&wj_wj%#esZ_4$x$BHN$*ph>L2!9kG_G~XZ`PJ?^y>5|P)apl!Ovfw~J(r}}p)XC2x)&pwe zRiSQsW}4AZ!Rl5uM)Cp*PH#q4}h-)Uu(F%Vp*~fgjgtALoA` z4{3fp^ZeG${@^2H*B`~Ffv+I`zk?nB?;iWHZ?9J0{-+I24PECI%0uJf-uUCbjvnB6 z=vh2Y*Z8A6aRbk7R)8=vZ9%89&)N=1X`iT|k?l)oi`s41ZN2Dt*V5iLr2L$^t?N;@ zcq3|3pi&wD?6xWAGPaK596Y>tQej1lv(H)ZmRj6Gx$T<>J|t0%*OC-Z614^Qz2x!)EKn+ZsX-gFU%ipIh3d8Q@w4#yN)~R0Xvz zrp=@S7jML9%bJDh=nsHa4qDh8`m~Mjf9F?)HLc6Iklgo^Z|0TE#JtHhc(JqQ(SoJW z9mPshy3Rkm=vsjdwl9YrQz?OW{?yL;|M=@->HS!)nvbnbAc;$&2Ws?E#mybBvdCLw+8Hn0o=G5|#0&3Sy+a~2# zeAZXZKTmq_GVG~Is8eOt=k083!Q4}~u*QjEh8mn$prLwN;YM5Yt!sY0M}?hwQb{lU zqF#e?f+d$4WT71?A(MTIcISnmI@9Z&NyA2$m2;#mg~?x~`{HJzWj+kkZgd1%^|2aX zx5wcQ`iM-qIvI*Yhz(bj7O3F|JirSZnhs?ZO&kAa8g#YIJ>WdHpuwY2u?h0 zV;B!+uP?LeG(tMk)Yr&UOiofOByZ%E%C>{jWB{os^Oma61GQ$Z-WeOHlemW0*^ zr+lP2sm6Eb1&;DJAmg{Yh#c>S z>Ev69e#TwA`YvKHWP2*zpdKs?ujx8(+@6{*asC&`zI}c9$Xgt!n*nR01ar_YUE zpR0bP?V=iAyrJhzZHE{ANJV}3|D$HfR%!n!Yv!*yH7j`Z59&)iyjHA!6WAA?AEFEN^1%_5s`fl4irZ3 zLNzF|tlZq6s#N`uU?HU-e=X5E$2&2k%MxZHH934?(tC^fm?^0GUXC1_Ej4x;-LrlY zbWMf$=+I;7BfGvN9`nvBy-wNZt-Bj~kPASTq{mx57S~KRzihVH`pTw2Ra<#${M=G) z)D_A@Q>fJalSV>Y^C#h9G_;iylU*IJBf;M{CVjk0yKG#ecG))NNC!L$mbQO#sIk;> z-vcY}T&s%vHmysLSqLiKBgyuBO>{fH{fWX7;4kqw` z5uskP{k?)^jS-|$rwMNzPbI_3s4A-pS4?N+Q}5ycl25wT7h_RZYJHjkHmTDK%KAD! zwdb?5ZQ4KV+A!a8l~aC2ZzOcUazXIJeFKjy0Oqb9$s>)=)lV_q`yKwL(?mt(108L& zji+|jF1`_zHD8i^#Prw&^q{m^2&RVI+B2O!fM~l_(I?LgJgHwk8{tcwgQlt>Z!!kB z9D`PV5c>e`e6$VzN&R-_@gJUUx+@t0{qqsg$A6$1@&Vu1Sg&4g#*mzf>g4L{Kg;`{ zNt%-fvs=xg24*amX2%+GJ4D5EZ1bUY>yYk74YWcA0QRim-1iQ~9`(K*59F^z?GPg$ zvTp|$zht=9JD@}(8-}wjTzn3N!p_Hfp&X~TqTjNEExG4W`y>kdvIGGMh+4AiN<(@% zYE|JvfQ9!z9mN3P(|Bzi&znboaDd8&b3pLj$^J764{StsbsGqVLK}=eNCK3gN)xYd z>(IA`$plumz~Nq7$NEWC!vJ?BF|g=|vG3uNbGgsNQoDb1s6S2K*!=rn&Ij*YoRWKc z1|vxo{eCtzm5ohjxILPw;!$N;7mXSj9^bo~=EaUZ{aM8Gt0>uvpqEJPJw(&;Jn{hC zxyt#(UahaAPw#FY-Tt}E+0VGdc&gW4s9M|+{=WK$OOtA@K6=^f15YGW4qxhC&~Eh2 ztX&PoPo^=n@qvv1bm;cD1)6GbUMldh^C67_CF`#41rdMEh8ek#)xE-TVG9w*up?aT z=f}pFyHDEaER47sxyITAKf_25#tNL!x3+VLP3C9H@Q zHcs=~Ihm7d;3;qa_~skP>&X zyiEe4(jliQq?c94P8b>IRlL6CaI{L`xaY)y1(7ev>N`Ai$6SHuijHIfeb9~yFJuyf zyR2TeUOSSlo|Y7Mq|qPRq}+e^06=`eb;`|nD9oWn(N|7niMc@6_* zo=xpGmjYr8bEmoD6deAFtEj7SoAMx`G)1?#VLO0?fhL}r}K#frMxrX{b{|ap?o?pH0!J-$r*RIL3J|n@f z?0}U~uwB0@n>fH`F5~#v%ced{n|AKCo79P@q5jQ*A%V~53>kSnFlS6Z5h)OK zf2IeJ=a2{1bbyIgw43AnPZAE6D}X0PR3V2l;xOi8G8B4gHWT4FYkb%{M6tAOq`LOr z@sKa?ina?n=&OWpb5of(Eubs(^0^D+R>@PDHQZOq`n+IiH-Q*AomQx#>B-4kKa}XF z-*BFu~SvZxv|;g;ut^Vu0iF9oN<%X@1uD% zr}~cj&HebG#YrGe@{G^eSBW{?34--)mpYRmt_+w=bd{*O6k6;Dm+*=^3U~!{Jt_Q{ z!E2y)Mq91N*b^iCAwHmQ87^n4j{z2n5EZUG*@K21o;+n^gSX9e_r1$lQ&^dL)s-J< zjGwi1hCLJ}yZf4kT!-h5Uo*V;KpT?Yczid5cEDzHLtrphIs-)Y2kWJSi_UEZ)*q*R zW!IuQ1_f!{X~&J0&7Pi z^6A&9j_ag@k8`_{_AWYTQOQeJ=cWyVyKwnUG9ffgUBl z73OZ`G}!Sffi8QR`0K^5CZ?NmWAp^GrI0HtyEbz5>exYd>ZG2H-SnZ3DKLZNN_+?_ z9?$QI?zPX+g!T=NMaTW!E_UW!yYM%!LQXx2r>0>PuZ6AQXO;y9LI^p>1h}h$FE(g$ zUd2&k-GG#sCgfgdo_$)SS+Lv3)uCEj7@ywiUMCVEL<0|yg!?X)$ zA^z7tI$9b~H}hn8#1K_QP@FP=9ITwijR;gtuaZq4H+82zaW@lNd1x5!bEKs=LxFbl zXkNATN)%=uY_}S=3`r(JfCI!L{3=*jpRq(#Sx-L}6R;q49ouiyY8css%)G>CV$d2& zNP-4cHH+lCggkSCvqqr@z`X?#g+!JM+t5717@s;&%*_7Ufr7=Q*zo=Z7o%n)8x@9P zzM)Qyq{SmN;6}=uv&@_9b2XaYdC6!Dm^Zv?9N#2qI7I5i?Vc z(CG2NHw_>bQXq`J4i-^yReVpCdRpJaEnB;nkJRs<<(=d%&K0b?T_)&tm|KL$zPB_) z(X#XKrsK}^+(X^xUptOjVshU~JZRi}mUV<7Qb zSbg8J=RupERZl-$Z=*`jc8~NrBgiTJ^NkTtylIaxepfFz%Ef9QO0DwT*r#%w#RWy719OyYeluW5Z228Mw~UF*IQdjmU;cN!{RxSS7;isDxxQ#!qAOu0DC zggEQOX_9`lR}vBlE;0mSUW5e0HoUJ>yXy17ep03Suj1T*7`VEusqtd5*olVYl?G#v z?lXNKNQZ`Sdl>WKC4H*VIqhikW1(uJk9@xeIA$#13WQ%=3s&J7%Ae5BO>N+D;pQv# z3Enk$S%z_GD5S1w%x=^EmB-o(4cAxx?9X-0je-?n2VmFugsI4=^5a}OyyiU<6;=`H zGhlYSGTo^y^GDLP+kdx?78l13Y6NJOG1mTm7h7gi_8u>+w&ca5u}DqcOKqoxR0?Nf zXY|Q=KU}<-R#+UlAi$mD9`5n7UNbVNw@X|Sv7KPic*|5(LRc<(VKBzWLhAuE5gFTA zmOrNS;tXHmH*%g|<88d^>bz2|$gowMA&uQX7%L92+3XmJf{Dm!1?;%7H*+ zlV+c~FP$#rUZB6jleCWPb6qtq z?Q^H&mfC2~sv1mjZLL z5S;BoAhjAQo1s;hnU2xvUhhg@nIjUVgOnPvC&CxjV@*+W+U5r=e%O9re^4+*6S40G+j1Qmj zoc+qBPe$MdSterS^5R$FHspwGk(d{D;jeF|Dtg@>{lM`R_&!f->oDmlGl0udhlW09-+!0 zE;AE_G5_mN^7$ivkn~BJ`P$tn9+`NJJQQu;CBJaR z=t>Xa=Sd1ItEebO+eqv83~DSXo>_l&p)zBjUeSDt8c5&pM)UVKmFF93=Njmz1^V_F zk?(Ex`l1oky$bljTcjZZ zty8SzyYOVT%3lTY@QkOATP((>w>gv!$iL>%y}2t*?a<$^n}rQ;hFapJV+)Ln7> zq3*rM9&uj9{cv8wYs%qBxx63U^!Aj6d^wodNOHetD|g<5>V$K>bM55%v+{e7n`E^@ z6yNrB={O$?6HZ^3macJuZLoNag{XnSZmL0_+cH(jxuAJ^mil1zavNT_c9Sw1?6u;} z_V+7mb>q~uRPGbuRKki5+*~XV0YSmZ+aNXo2C>2tNk$ztCMS(N6%blj2QVnEY@6#Cc0OYpK z-s+>34hf~H*QA;;ZiWP{w*hJ^)+y%y%b`u!Z=g*8tG(wJXtTsxUxEp1l&}63C@-iiJeZvY_N6+wzuuG;if*PACL{$ zxwLYP(3lh3aj0iwV><^_jO-On;UkxcufX(QckB0HUu@Su`tn?abZJfAk-rXOg78H2gG?b@@K$$7pCc-K5l^byE*AGDO~OIOWE70#|uq1#(s96bW}H&b^yJjQ#b zd!fBvPq$h-|LsiS39@bqsp7|HAa4Q2-YyFbD&#zmP}V#Jz&8m) z#Zg&`KpkJcuI59t@0l(Z2z`{gqBu?$QEsdaOM znRLSz@j1@0IGUH|p$a%wC%mx*?vc+r~ zJzlnIMq;*@O~LlAUzyG9Ps~Od`GK~_po>B?+z68XL02A*tA{KtMF7Go6^E?V1~dlK zrM$>eLS|9LF5HIZVO7$`YDomAH$BsYD!eAv*${?85}d7jaJg=MgxTd2K8{JoWbZb( zI!GuZzrkh=UBu7`RKht|!rZ6LaRajsfpmia@WpGk!H8afAK!?->aF)=eIw2BJy}oT z8e-0|#|+uHOnQ-Y&AVo-M*mcNe&{dvV_BQ7@MHwBglx)3KvTVgY@~|J!IQ#7#nx(UZ7pZOx}$ho|m zqpdkH*`A`2>-+P*qzG^)mhD<9DOR-#I-*wL-5#5s6=j!M zNA7Bn(-WL?&0m@^{oXFl|Eb`p-eXP!n%15_Jk(z22M4Y`47_gM@t=TLW4W zyG6Gc9{amp?AuEWnTCLfTei;B1LnNRkHe+8LSom?+>6?{W!5}$5lOC!)7!QCCZ@v8 zCap>C-e1{V|1*C#oBbJf&qBRSM5LbZrGT>M?mHv9y2Pc=q@{#QH(g9q+WVQE!*iPP zeD0Zc-{Z@v#{!LkR!+PEn$6lZBCV>)GSoWIdWwCtUqC^f+Qo(V&W0Sne=}>-kf@iQ zj0hY^bSl*qTX{WcT9cOA-<5|;Gh;#yn@>|JszW*r3>lP-juJTb>Vpf;3u zp6I_qZ46mGZS3{T|J~T8I{3HPX8pg|=D*(Sf4$fL`+oX6*yhYV+)KQThJu?l^{|^~ zVUgi$piQ(8Z1}1{PV87`-a=dMjp^L!p{slKkDl{ z#Rs0@VLwjK$i{e7ow-*xGWnaN4b8#%KM}ME4Cy0I2No;^68)GcHJx>DrGlzZ%REyJ yX_yJe0Pq=-!k}gi%@IW9%n9#KZXayw#@5DA#|~ZV2755=3YWd=uS=i!$NvFRxf?qG literal 0 HcmV?d00001 diff --git a/opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen06b-fixtures.json.gz b/opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen06b-fixtures.json.gz new file mode 100644 index 0000000000000000000000000000000000000000..f227ba43e5ec881a13aab6b1dca00c2d15463f67 GIT binary patch literal 24169 zcmeHvdtB0KzW+HpzujiG(wb7c%G#7RX`1FG74b}t<1|TD6DoPjyXGxTQAA;PD=jH4 z$tx;rmX_iTOjJO&!AnR=N?s6=@t%Sjs30Qnqq#Mmnv1iub9T?`Jb(H6`hEPOeE2-? z+eZNU-T!#Gey0-hywD%?7eAkuC#4-FiJZ^(B5s)eXKKO47YyA667fBGL+8oU5Emxz zgZCaJdLR0Of69fwTkrkLe>vy=*U?OLc#P>TjYm5Afc$8eblqm6>8@lkBIc_u}K8+X-d1#AkM6&&h%#oVxKzPXo2H8sviZ4Dxu1$4>^$GVfE zDZSzIHo_B@bXg%W^04p^)Q4W5X2!Er^6@#|M{xpKLTuQ!DWc(Y)?yNFO3)s8kwOD` zlbWrs>&RVjmD}g4QaB05MRmQc+7(OUzVkyoM9mTR)pO(vHX6qud--6oaOTnI4PT&_ zN#?}pxOdOKYN~8yDZn~882i~=csz}Y`ZPTi!FY7nBYbXy;L@?4gDbO-I~#^|qhiRm z*1N0p`*RrD7kUmhMK^nTQU??i+L3q2_Qe|@rznfzjSqn;&4b|FcG(+{yn(A)abGA1i(A>DXWQ@6RwLi%8 zfK3ySX<}NZ2g-<`z$wRy=8iew(d&xr%}inQXMob0-nhXOo$Hl*z^Wnl*THgkitpeFrtpHzLH2Zb zHdx+VQ3}Dv6K7#;27H;C15yMw%vfS)53jO)F1ffOhjq?*uV$cRr)I#W*LhdTjr#{?=gl!+BKl{*stmsFj@C)N0%ATnc0K3Z?PMX_D$Msa`=MCbJ)3 zJ=%|NW@r*AVfpn!0b!QB2h2p?C!xgnLF7TEW?ARc_*4gI00huf)tOqiV}@Qji!M(P z_2b7T5}#{ZwM8bHC9UXGCA3T_X5u9M8C}ZxOD^)krRO7bB2K9U(2_13w3gEiGL+(I zu{3qQtQxq&kq6VrDluEGoBZO`O2^pa)?|P(+IFsb9+&*$#p-%4v4>NQiiW8^qD*t5 z>_5|0tSUKldz@-%@_D-h} zn0O&K8w~9tlv)pg_+?PR zSaMFfMdZS`?e`w6yxiX0f_?(@&}gU5)q9T=_dwOxjhotImc-Gym7!c>sg`9vpasW(<=g1Tv&mDpmet>? zM?d~@kNB|H$F8bIom)A=ss|O+{u#+mUV;qmRgB(<8vX*~AC0{%g2Ig`{$KcCn@rOh|#CrQE{3z^guBV}jU zV<6YtA?3vqHH9*cvTgoy2VrHA>z(2w%kj;RpOARe)+nU-1Hpk?dg=>d5pKFLE~4Ho zJ=E2zSMokVAg22cmOAot*uK)4T}SPiAg)D#WS_T#avX9EsoQpdg}_RToR02LjI zC}9P2=l@XdYw29(1>2U=qN8W$=IbWa!X7hWB)$onTD{3<^?iae(d%fRv7W+K+4Ru))Tij%!> zriwAqt3WNhZ^k6$+;m)7qF%<^D`LdI&$*dHacN<`s8G@LsJ~9>8pBjOc!cn=&ZK{oVsB1YlqncMtRDx8MA{8e^ zIZTo(TFbb|zT=NqJz(#_Ie}iyQf++w##4Gq(B0%nnopxjkJFI5YLgt-(Gk?=#ty}4 zBsVG24sWE7)IHa9;4E?HwR)hIw|$@Ya1Mp_lz*s6;xkI27RTkr5#^SOT{;+94$@S{ zR$Y{+l+xhl+$Z)EerBkN=)Et$i+8u$g=V$~|8E;@&ENF)4_% z<}?&6nzZW3a{S~ObcygxD#W&eNQew}FpVfpb+CR`T$yUL5zZxbj*Lk7nC4l+CUttdP(=ln>6e8+|hRA7u#T3%tX?4BLsMUki<#)~nYDu}V z^Q2EKMIJ1okiZvT%2@MKZ5){gzdtG*`vnWth~pk8$6rs^XWsFvpSn>Wnt#0h-R`uL zJ~3?_W<{okmlMn?@o9h1Uh=N)7;`u<3MGxQb;UNI0U*c`dOa=Om?b{GA>t}PU^xsN z-s>X0tJl~14lDS5Lz6t9P!`%YV-rDN-!^9#IP+qtBPePD!;SKJ5?$|Jm~Lh1MCVd0gd8ao|lJ+DdBYUir_%kShh_-s_*_(Xzx^N;Z z#N4`Dj!C>NqWno##(IZf1^sj`@5-7hGDD>`IRysRFtAL}Wr!LR;iLzM%U79ZMSkjn zlBf&nq$5=tU2`BqlX$gky2DsHDGFsC9lxqu3{=((6cuk2jq%&C?bz|@BJ-A}gfC^! z{3rGiZ`j$JK)uQ$5*`gu%TDbs^56(7lVaqv=wsd56>L%4t^I9bA0cU&F9ga*pL*cYgc|%BhHF=|L17MdFHBvV^n2(lmV8k{%`~BWzHB`51n-J9ubdJ??C7h0tx%0U<9BABC zVKtyo!C$Adv#O8jQ4vjQ8&CFobMey$N>Vjh$Q3ziRs87TxKI-ZTw%5|_<&|yv-o-N zJWesEWV*~F79TlH8)l&HCPwp9QA6o+fjk%hjGb$3d1EzO)IOgo52h(wIf`{6&C@KT z{FM0ll$52bl&w~40)Q)Q0|tjqE3{i47b@F>X|i6+7j1`%s&p7J(J>~|nf=2_ibO&| z^Wi6L<6TL>!D0Ko?R4t+IdMYHI?Qad{3U@XGKthDkumI%^D)VJ8{*vzcz1SAF!6U91;dtg?%9yS_6EW= zgX;Zilv(Giy|JQPzbK;k0JeN}@vvx=*9&j7(rVpDUhjCT57Evog^L_v?sf3hT(86= z_~6I#u#P%B_5By3wJsWGmBMIVE^Fv0zt1!o!$3lBrBF3LpP1FHcp?ysm${fBv1o!2I^9^YbG{%mf&I)95u~ec2wHvZdi-2t277osd$wY`s}*gWM}# z7BP>m7T1KF$@cLJ8FMNP9qSxZ(+`&&eqO*ANqsfq%xhm1;ZEbL*p`%Y!r_HR4OV5i z#1+)2khCTRp?jwj{4YKW476y>c229BKC$%e!n}8$l49SLaw5 zOl&^50%dC^_s2W(UA?>}y$)Mz(xeTlIX3l4Q)f*@{C9kGq(gFz&!9VS}ZbxG(pUw^Lh{MOhuBSF+SmSXHB9i+j!K;k+=6u(_pakn@dPj8a%Xj zA3j?c*}fGl=Iu2hNvWpS%LI-D&w`=jP2&%(d}u{RW%C<|rj3UI8TC?4hUkKcbuO@K zmven1(l+LB)f5`bn4KGsZOHfZ(O18uW2EP8vY6R0;su+6`H0_XXpP#~El@uJ}u>>O#MYTuO0D$_fspTcTt6;8H-}wcbT8DK#4)NY zH|tFEZA$)gD#a+Z9*m_Fbf)Ovrr(r0dI7SD&ESenrJ+`@*F$wv=VU@d{60@c&z;8H z&mO2$g`rh|yS z#A7e@iIJGXSOKSdRdVxk09nQcFm0%B+vc+uRPWkbgnZoXN z)-BW2=O!<5Z(*bDe1@DO;|hi%((G3tTSexc zoQYu|C<1zQSN8|EgJt_tEL$>AJEL@?GRqJ>n*^hOnfe@^mGeYjvP$ufE36;+y}z{R zlemP)Kq5)Zm?fF$+j-zmwC}{07#mhF9tQM~AiWPGZe<0s?{UJ_K&;5E(`c4hk1KWL zr8ZTU6Ar8#noTJ+WFNq0CquI7H~TRQFAy2>c@U3@=WZach+^zw(L4RE?6y~3HiVuv z*IB5Z4Z4Cro}|EDHJGvp((J1SBP&5E7VMGA9P~yv!9-5RI#KWYU0GRH=|V-CS7%u5UyUjZ8r@?_~)VYocT^aYE?X6F=g z1tzll#=^wtOe@)UN-@_-jPhCu8lxVwlnkF2g5_&37yie{$Ls?)rqGXPZnhBt0g%nC z>P>)iP}ex!u@3zr${i!=7b0j{3Iwu+Rev%I3_-5sC5Yx`C_}ikQFeU)v)*CgCMdgw ziAzQ~V%s%2_fm7pz_Hr-W(UJe{@>a^9#NL|Q3MmEUKQcMjF`E64s!OcX!`LWh3B6I z_wsH=lD7d)hoo)l7wdNlmcxa~MR@0hWzSi=g23%A=~_pDQGJp#A5YZ{+t&d>e8<(= z+8;kgsTq}!ulR?-Mm7I15X!c->*Irw${ae`R<9&~s&EYv3ow;}mr!hi*| zK3RJ!G*2V$dC${b+FtpA4sM-CalPQ~{#kQ-qm;zg`Nw?Ug6MhQp{`&zf+5!+=%Q)$ zV5D|~LQmAR)agt4N+XVxJmh7wTs7j|=;6Cocx{fz+V1mI%EKM;()<*cH_uCF;l=g> zi#IFZv{to(d!{@G4~uKsjR|JNV>Z?QFFi;dTQ`Bjwf1;-o99HFN?w_z9(K(ZU5Dzm zU_t{S<~8@F{|}-IIK7$Gq=l2*>=!+n6n)q+O24Xm<7@R9iRUIPBw&C%Op3dXCJ=WF zPYDp$OqFe~5Y%ln*+!P$en7uyh^1z^MADK?+c|!5Hp2tE;-FA))jmvdNDAJ*U@7iX zVyN30#Z9ooV0|1kIJLNArdfT8H4D>vkH{R*P*RaM6uLQv(XsDu^ftIQqYf?SS<9l)Q_JW>y{`- zl8BcWV~dXh$QeJ5z7CSq=u7t{`Z7?XuXeAi`cTCe6NlZ;L;&|W zJti{s`BydeR*%8WC7RJuL08R%ox;ty`*jC_7uOEyMYJzgDc9Cn!Y?vX@q{LY<|@!F zuhq#jTeS78UJEfS5VFdswrP2%0tkAtrzpjcJcMm8JBWt-Do5$*rDU!ACQUj%vRqhG zPJgNu+IKg?DuMvrFRyyuQy$74xv;(M?w&R1o<*lx#XV1GKy!F(Jx0V=^ayk-?2dC98y& zQoE?RZ>#~5zP7@1N!U+2LRS4r`O(Zz8x4bh7~a;#UfKgSJ~6k?OFW3J_HoH($ER>| zhw?Ih#!sP2UHzj5*N*8``eKJ7*HEC( z$F8a7MT$RFq$MD)EdK|SP`pj*H9Kfo zSkHBH#7*cevP1jMfKdZ3n#6hrJI3`HH~7T<8N9PRsV|lT>wH~y8Lpff>#p=IU{#H~ z)w?{I^gH{IxA%hFVKh#Fr6>4%>T05S<6-23xXN8GebOU~#R-ZXNa(u;-^*vVWj>B<`Duwbf^0=Jg$$sPk|2Rda7J3S|j(-x{mq(G%wgol8GT zF6RHPCKpx$HM!V{#V#M@Bcj42$9})&cRgvfdT6g+*P68y^#)~*ys`35N5}IJg3*DL zt6T3Jt4b?v3`alS(Q&-X9olv{=*lLS(kw{#-|wVLca#G`clQ*<8G>f@s>d#Mj(vW6 zcK2!X17CT#|IjH)$%ic+3W)Gx9o@e&Q}da;6!1l?O$D)vHHaL%qGvCZ(@VSRnQWWt zl#@9M$u7;M*9)Rq?c|P)3}>x=)$yFe$Pb3^EQ!)yVJfqFUeGH{mHo=-SZXc(+CQ&6 z%qx0(Styo6K5gQJ4}^aO(i=Ptc!@A`n%!PFfwwxH5m@N`Htc2czODP}`2C&n_EEci zUMvjE1B`y6aV>Oe=x4CiIvVIJBmM_oke|VK<+RCjS{XKu>Yu66l_?r=gg`0TPG((zS^S8uL!B0vJQ&8_NK18 zSFP(Vq}kcO^HyDVn)RBl>l~b_*d(TO5%j}4Ky`4?V)19(S=nO}>j|A+&g!6qil1(T z-g4m*a+d-R3&hjDR@j16K4J(Zo%Rmm1r79O(a+zTUQljL{&(T&Fo1Fkjw`t;Kr*a)>@^brZ1Fab) zdE_wV{U&G=dTc=>s}dywp8_6glj=`PLK0wK1`??fj^J3Tl^2gn?2|JK>{hv)OD(q1yFaQMo{(9ej6AL7iB`RPN~ z1+vkgSJ{vY@p0dVLlUfn6h0L$sXWdPA8LuB4*ug^H zgg63eN0a^>2+*86W^VMl+c6JK z4O!*XS{yu8%}Fyj_uAEm=~n@QLBsz9taYidoh+;njY_M)r!}pbfuLF1p-DCb%P6UP zpm$DAHQ%@`b!+~fmWPyM3pSvg2*7?rHW1{7&1O3_oSD}!>>s% znHpYDb3I&^9*cX&L{l)!E{>>Y^%-Gd?8L+)75Ps~6;GYzP@gC#%UMb5>0K%2$H_1A zFYStW++1K^M2Z*HgVkQ*K|g}?I#wNS$Z2gkPY(T3lG-BZ`>xKX`(Ylh>U<}uxAl{6 zLPue`Z_!-d(ht{osW$4u+g;PQ3CU}^CQniA83q_dRh}UEm+z59W(ITRKhc=KL6mM9 z`drT+1C%!KXaXq}}zIlANE zB>d`5y*eE<1B8XX7bNTJ&Ect*)5+By~ZzpovP{u}I2<%<^dFG|*G zwETPAuR5$g^EMj&kWWsyk$S}jodMk8{tQ7?dzuffx>|@rHQ~Df;kvjxMxQu3<$R;Y z#(lpfsAG4V{iP>bBLnNmO((c7{#uK0$+H2H2pkXDxPYZ9%6U%JxyG66b}O~nimY| zS~9xDcU`gt$mZvL(#+`?I0bIKJ!29ZP`fKN;JYA2!`ErVigoMP*f4(H?X{?QVY`z< zPiGK42&4W5GwBmH(|yDe*{t3%%KGypZ+c4%GxeTI@5O>A!DB1AoDxbv8Q+` zQgu%B({#Xj>EF%}>y9-O*3poo)4loQXSg zEzJ>#-vF+=L`aiIAJs!=-uV|U+Y}!Z+(z> zS!_gWd782;iu#@>eMnt-e!}4g1C`fdZr_-Yp=Vy1kkf@2*=y42XiXs`j@Ju@TEI7J zRM3%Ub-QLl^T3+&@`x4vh^DFni=~tCaNDrK$i6FecoOSwbN&y5bZXCKODwSDI&X4W z_K|4tJL(r?8~&PY#PPAWk<&DdEj=r9NUmd%ofSOc@?;{s>3Q`&Y{|&Dq~=(9wm~w% zQ@Y?tYR+-aVUBe&842e6z37P}dx;Z%!bvpmG+|*M&+lTeyN{jw)ocHRm2B&up$h`a zr{`57=DPk&qJ5J1xe34kg<3|gbd=zzD79v1Vo#m#UI(~W4OHG}{7H`RCTm#h6``RH z8o6%%n^^Gb@%Y~~K*BiAKh~vko>T=s=b<;ReTM0(lwd}CME=fbu9=D?t#BwJ7@ma-=U=C zyo|?`*j>hJgl;!+bKDP7d@DtQla?pEj|?6{t4+PnKdvYzJm_$--(oWCRT$;cMWd2W zZVjczQzG(pd;)$A`m|QM^*?aGO7Mh*4LSGH2$gJ7=ZOL-o?69MrO;=l#rPSZBny18 z)_uz3!m~8XCp~*^xH$DqzCG4cay_H=TRW1|HhQw%w(`zj9I5dk5MA87dg`I&d%SN? z%-c28x3y21=Z~~cijW(MnfEpi-Tj48fI>2T2u!9#3ZUNiymES5!_O{F;HNwAd658d zF+TsmvpA^L>!y1msVKvVloAahzh7lu+^Ns{lRwFN81R0z$G1K~c>Sf{K&EetqD8W`H%=ccLsHl_6+zvfe!OI#Z=`eSk(oT)+ISKQq6@^H5A#+^#3(n+{%~q}O*;Jl>pt z(+R;OJl;HdVqKTYs zgPZJ?-=FnNQG@vFJX60eCUD@nA6=Ly)`|>?uLByoNXAU||p?Nuumaj)w0Ve$pqy|$M8#D4*!_VZ5 zZ$u3jei{5p`sczQe%B9<3;?|0i$p048LyfsliIC(S)!i(qnS!an>AGp)cLPwKA$$~ ztVv1#nK#QjTJP{5A-=VydU!-W@-}cqrXZ%GKq>)!U50Y;Sd8RFRInz43!16sCsKixJt3NU6%J?Hn2^nBF7|&{5AFinvcOva zNOgRLb=n+Xp11twDnKFP!Cv=7HTclc305=hPUl#vu!qiM>)Wgc z=Vd?C`Q5)hUtaqwRQBiLth)bxvkbChRxcp#Q)WWrfuoz}iTC}Fs*OsaA$HQQwu|)T zeJyqT?9(y!t%u*AQ@;Cjt6tRvIKN-S*+1rQ^sQZ3Uwh=4=!9B?J2ekD`%k2dP)wiR z?YLWUyr+^lT$UpC`HGR@3RGtXrk#QZ3Kx7W>*7q$EIoB)(-&*_AULVr{`KbhY7T_+ z6~@~hpvHKwDiL1s-q4UatAofh1PXX2Jgh!qRW>NR>V2Wgq%3yQSZ>Tl7_QYla;)Usy{BAG>g$ zxcs<8twR~y7cbvT3xRC;8S&>0>Pp&1jz0i1Ea83q?>Bgej%Oo{S)Icj1|&W*aVnts zwUOKkK~Vk`v+9(mrAy)Dg}NC59?WjWSr9`pIgS>)PLwT4LeNl~!bxbZwpX?V$Ev|oB>t~{CCOoq70v)KF{MnznKbgwy-10FB@>iC~SOUXOt8>!I_jrbMz!pJ8LjT&?+k&P6V^(InG z$&ybJ#8w^$NyJMp9ua-ttxb!AdLv9Dv8(4dku3`uGZm-2y^@W~fkBL#{eFX2YsowM z0(pD0?z)#U%Lnw=orxQqCjjm62Qxn@(iTu!{tdzqYkrBU^aZSWZoldF);q^wT5+RS zBaAZHVqkR3_$O)l)-(4!@Z5mgt+5_?p!;~k-pnd;?m>C zOs%c8!8`DnE44QqdsqvXw_y-?@nU*K`SES0mZ5h}oDAH;z7lt9I4IBgp{#=HxH2F< zlSL`sW8&@m+;2rmjzu)xdn~zY+Iw%kzm~wPbc|3`*qZ%9EVc55-xSDnyHUW z6W^a9$K)N%Zzz~=E}PhcwR;b>Qm0}pe6Fp4HQ;f0D+l}G{1w^s1v}bsn=y|1H)n-p ze8O>BI4L>A%kMLwH}DRUhva@S4e^#nrM~b+#b}PEF;dal+eYkM86!@|OwJ;sV-Q3T zAZMK~0$4D-y?!JixoLx-GsZ%KmHRe(>^d?yUF{k-y2A52q zY#Z7CGj;Ga-6r8(w?i&IA4U5pWCog=U$j=!;H&a#I!-q^o3^x^it|!WwqsewJ7xSIfwMN6=%Nx6>1WRKUuMF8TcM27%0p?)S%dH@0YQ zWz&fi`N##y;$u2zPB;-c0UN}(aYs&|5y^Zst~IEL4j- z{^#-&aJTA!!HDd*rlTZZOA{HgCA+t%nSUkP7_y zRonlSIXacoq)r6k(CdG!>AXP9`{hj^p7!ajccYgM`~~Bhh=%6f%lKeEEiwb51zwtw z@eWb%4P={EYDKu3PqXj$An=4e$*zcMVtLbqeD(}Uj1=WnMYc<%Z|^znhvS!_;cZKNkf5idRRUlT@58mlOsou zV`hiPi7Wt!cEsaG@D#gpHX7%)B>dA8n5s>CfdFWN+l?=@&Q+Z36g2D?!>%#|Nk;Ms zphR{k7J`x1y5C*^!k~vYn4;0;xvw`&DYiFuRdlFQyB{rx5Q;hLA3uw7k>uRHh~?5_ z!_MM1jKN6HW2VrIkzr0vZpy^OWK+XnY&{f}Wa;IdO!g@l8pgBQCq5d`)hvroi8$ho zGo~jJJ#Q!`k-63V23#w8EM+hy+HRw7t!^38!aMzodq#_6lDV43=p$+WgtW)CYl`Gn z;M-<6e_pQbNA9Xci2AwB@gu$9?d&PU5%at>W~Z*4NFrfwsFkrr3Iuw4O5z5A=H zKspdzVu1Tr(}}8HvBO?LjvpIbh#?SfFTCk*UwgR@d&z(AY4Fbs#X4Ep+wl3P) z;KcKI^kQ~LKsgwIFv?#HLN9KfK#*?ZJJ4?1nsS>3#=N4)4+(am&T-RUs0+!t7LhR; zV`g0yN-%B?njLm4y=DTgM_^bW)bi?RxJ6N+3#q<6*}Io`=@7F66gG)A!v^;fgyO(q zsh3B+6STLF17zvc=CX2}A|Se1XmN-&KwO>y<+YjN02_%0BQ7|JJsjC_hp&;dL`0J9 zw$J&PRFP_#!6sWjr`a8wWMAu?^=-1KbV-ROdqg+RwvJzz4|jK_i({tbQL1(nQpIoP zdq&uC@%A9B@zn5~jseL^un-O3RNqdYF7(}*-e1P788&kBr!@j51xpM`=b_!z^}e1i z4(|B&2R=7)N-kVmuV|p!hDYv7e+y9g6)vITg#3)urzYBT*v^+AC4~B(A}E^DbolZf zVZFVo*DK>~1HeC@2>!sG|Hlv4%<6u<4L`sSk56TS!7*oUk){>#rXQYR@ZwDhu|;FH zmw%Q)@b7k$%4$p}qN(QKV8FAz@!3{hpQSMvwIMy{3XQI-=`yk?c-r%?EVgY`xp5x4 z_SlxVzaGGvzc1KB?KI0fzu<;IFV;?&M(>Tf|M=_fr22{O)PMhnyHl8;6qN7QE57fk z+gtlU9$4?*=6gSb5aYddi85UQlM>4Bqtd5vwL{@Tr76Wzo4UWbA;N?&y-d8>{pS!Az@dngJgDwo5X z3gEv8^qBlpK~J)ke+Iy=X?fHQrDV`AT@VSPbX7{hi!vviR+OD`Bv^-*^kQ-$EW=(^ zo*=6UsAb{kf>Y63=E;8r&5rM7B-=QQ#wnIZAJG?`;!ThuNWyB#I-Lxp zbR6XaN?PScebHKC?+sj?9DUTUW3QSveldKgFsiLa*JVBxL|IUht>}r$?wZ^$c)hU{ zGUzg91bL*u&Q5%x-WJj5(O76%A%ZoqaYdR|Uv$Q8n=34o_}f?ZjosipwE6-=A^mvNnX7(r7dA&8p0{sVjTg|*& zTP>~sR?bG|OicSm+%gx}xlip|yS4SNzeDN6%y4uKX1r$_<>z{Dtm*m$iGL2#3%)g4 zUJs6<4X!u%94>DBZ}jlY^yX@t$as5r4k3*)>><|yxT`5*mp2^c*rz}B5aMc~-1omS zy`uVH0c{E0hQ@bgZt)v#!GG2sx1G88gmCOV2yg3y>hs(R-_M1AKV~Q^2oB5!rBe)> z>|7z{y%kgO^u=sHkG4A3I%xM@my*xsYdg{wp|jMk@|n}o;XMw50AG%fch>7Ji|kxF zxM;f>t)INbaYZT4y@?@zC?3^HF-|@lkrU}c+R7<1FKvqT#rt`c4p5zvyoa$~?GZpz0aUN4Z*1+{3uTTU)!$bPrt3a6o)pT zCqstoyu6Ij#d8^HCB>4de ztw>E#*=cF1X@ZH0!W5w*Dk*tDL`LzDA`+r_K;WZhX3ey6+SAPX=K9w9_U|vRm-ppG zu7~@%@B6ubF95Up53kmrR8u@2Mtl7Bw<`JaW+#4ASI=qjF8@ER=Xf_jbMW~Wt%dsv z>&p(a3r@wdy^A!jKHmDtmb_gb{fFOw{pq7)CXp>)yJvm71`FDqyR1Q-?qmY@0&QwC zSk|b;7dr|j<0n1wrPI3i^}D4F^Nl-m20Uam#)!xMMN`3pZZv1DkSnFP8Y4{iEF&lQd+BOm+G7G+a;2-9ix!9D zv>`tsq)(pAB{puTnS3FK=##rY~RHYN5N!VZvuSSLt~+0}TXKrBsE^o6C*_cjZA ztWqR?stKY6jn{3a#iCihg2`3!obk&kF>S%m>0G*2|^-A^5nA1m;-b8pypwM6Jd z@X8xFUO)243Pvq3E)gS6Gx9~#qzny)pt+X(1f)jZaY@g8&!!>NN#I%pcyHkB1Z-5vGPfu$i5;4e9BotWSM#^m{K} z%otOjK1*Hz)=0)A`p9~NbyX(1amu!*eT}Cj18mpcP%R&`npAVBPJt(9>OtbnoULYM z4qavu?|U*Ep8jX2oEKRZb69hU-KNrXNc_)KYwVPNts@TN8b{sdDiABJTEC z!Ti}4B6G0$)Q(CFON!vDlqV05?odsI?@8dt7Vbl0>1&V|a_%BjIvV7*VN$jj3TqZa zldymwWUqM`r#Fxr>y(?NI2o=Qfj(0BWC2d#Zj0Ba5L!y@qr7n%p%%|mz+6p;WLOMs zX3jm^i@3_04Oo~8_mW&TUE&F+Dgk%}e-(bNQP@dK2?i+?Ak^Zw9+9#X7l2|EJ0jKq z9fcuhi5eZKSk-7rhTYOj!E9#pbxgDUk|1UP20AEzIkae16MUBlQw>ikFUGNnd{iEA zX;DE;LEEcXM>KR56YPjLZm5<300AiTWup~%Prb>vLoX;IP|WdpfP8Y$+|cahgPUGGU~<`yF* zfidpY$mR77vkNtRtwM!Gphr)3;zea4%L%$UdJ&R`uRcl}+{577x#skajtWT_33Ltk zlVOj+JIKVRi-XKUXQ{cgk{kYvzX@#jU~u7obA1t1mY^~p0#zEuSp5z z+#EGNmrD}~lX6F$x?||oA*H1%H)XVzcR+HbYzL8{dW2~!bHQvg6qmEq4}FLxZ6sxZJKa zfrCeJ0^g0YMe}+lrX~D2XkGpEd7HLkZuns+F3`zVgbVFcUU=BA%;^bjA)B?=@atxv zJXTkHPp0=Nc;!wetsyrHHnL-XVz=h?4CN?GwXg`YpLYsaxQb~J`Hx>;P`^aMw0yf` z=7RSjS5=kf_3WvN`(=&ulQKZPWG0)MTuy=ZUdudp6n!f3KHd^hxmp_P5e`ET&&oA= zd|QHumN<(*pPMP%ar?GnBEqJfSgc!W!nf2Yn;4QwD8k&AiFX)`sL@Ax8^Gfqx!qdYm+LQ;-rpqHD@sb&SHaqDtKCYrClP4#WX zeHH;pjA-ww`CKmg`dwc!?$C0)Gk5r0#5yQBd09WnWpa|zQjgkBsQ8#nzEuO8U4+Xk zzdCzs9Xm8zC;9d?n=)ANHG$9j?9|xkvt-%%gg|Hq3kNL2|?kN`=YDy{u| z@4fasnbgN&j9+!Ug=r$3Y4m*9Z+YnJ1-_o>f?UzB0NhB!Fv7cHOUmIvNf!CCiDxBT z@j`a zcOkcQwzQ8^@|-K-uv_ifTiS@Z$Y>0r#!GVC0=Wqj~-7T9Zp&J^k(-3YSpL84|OEqNj>Es+@W-ZB~Sj~ zIyg~?%<`6bqFzegdr@~SHy(4yOG z8t|Dhf`)Imh$+>K16tC133R8RoeRO!vvZqgFW#JNw6MR3F9 z@X_1sfvThS*bi&LzSq+XXpZ#*d+`Ojhb(_Fgv9*2&OJ} zTfS^EE2z+9L`FoKPTc4nOjN`Z@+h`#O(PwNzs%L<9stK4DdE0DK8`jn);k>K8N=8qYuJOop15m*&RRd|e7JoguQGVq)FKM+eVr z%APXWH_h+6R54PizI+BhY*#LKh{gg=mBVF&j{btw7kNDi zDXycEMU&xeHN(A!T7(NZErM~=HIkGkg{NMjW*R=HIL#_Utv~ESFGowhkwR&$$Mru> zKz&|Se}en+zp4@{$8CTl0>^W4L>Ox_cEGdzM6(vI$i%3O@hISns4#wvlZ}XJ!dy~5 zobI<_mJs#w*R!EkOkwffNg!@6tdo_CdO@ZbE!9j;?ic4WkPC`t#eBhZU{6@GccUk( zxyX}t5bynwTa9Vkn4jMxp80vX<8ZVPM~i!{ZF#fj$b0b*w5KbQIR|ISX}D#-qV9O) zIg4ps3l03Ow^yHLy7jpgiqucF2)eHrn(3L0vMmm|AvQqPB~7LI0=-Rdj1G_ZbnU8a zTwIz9uuh7-mBz?ljH^ycJo*Z8ACE+geo1JbdnYQ!eyc{s_E)IruSUh0##m3Hi{ZEL zjadS;NdKb<*{^_83ALXUwi)7j_y{&I6n@q9XoB@MH6XKd{Ifc8$%%N>!k@Da(7Xj}gVP;XcRvf)_4^fAYB%&G#`PFJ7( z$UZ6iYIk&hvgTD@0^|8mU1s}Ws{xF9VP7-HFWP3XPdge41y}6C*4wr*Lp@nt=9OG{ zmMbJDh;GKSf9&X!<;X1t`*Z|3Zfx~!kr!5IFroZn$ZB>qZp9}0mx_SUPOrA=8rRH5Z8lw8fxR*yf3gj2M6-W+cdqrD=ub@)#56E$S%F$>cIDqBBi8 zIy_0=OOm2uGp4!OgN%Wv^KoKc^VBr@*N`V2YWhX!W)uA5!n1;pEFB8GEf)_roYHZb zJT;Z1G%Ps$DT`ZsEo)*h;IBT`?+uCz3ow7w;L{Iq4RuqFuF-Pw0&CrMb^FD{8;}C&F(hQ?l?ByI^xWqHKg)fIb=aVNa2^c;7G z5wrL>k#llA5m=cE?GFwEfid+O(J)9LWY)KT9oVfHV!2mdYMi21X)r!C518jC)MrCU z+hXPfqwz1cST!Ny_j`ThMyL)k8G~`#Kb~Ly?>N!YnAzag>}Hyul?xZa4K_+4m-$+* zS0JQER5Rn~{q>XZ*>cx0zFjYzCC{{@{zF?u`5i>8w=UKYS#bpq{E^fBO z1*`T-Y!hBia1<^bC}~;Px4#I$BqpzT-Tb=|jP?q_h-w5QN(OgoC6Kn0Ljmu-RNNK~ zq{m%J;wMm@t~lAna>L=p&BGrUGfO-Cy*N*?ZZ}7L4%#2@d{}yRn6ssnaKDvhztMEi zJ3m`~`c$H9I=8+bw`#|^+_*)lxH3e#u!lVRa;Ex9hr{XOKvCL%X1woGiN?mkvQf}e zh^vz~YBQkxbBwvY4ixuuU6h6Wv5PX)U39$tyDlnE$!Z=aUj|ujq3S#MG_wxrg3!>* zpiNuST4#>04O?zAe~p@y`Km{Tuf@k5-_c#tOLm~5AH+xRg`bEIcFvpPgAEUH$cjzz z?(gMlI`Xx3W58Kqun(PFiyd|9&COGsvV%C!AjD_E(WhslZQC_4w+r6HP4ZkKu-@XT zl#@BtD*Kw_)mB!6`w)RX+JH#9ur>TDQ{Y>LO9~^65;I>ryeTMB#B~_F+QT&bK@+d2 zcxdAaOa9~?@)G1V=#h_T9)Zp%c3u+szZ(-BBY^|Q*Xw5(t-^*?@Y4sb3KxfiBFVPa zIu-i8*^KkC(OrA%BPd>8jkO=d4vG;yN%Af&9Bv^?+{n#M`r?o8U4;PEIjduriSJu1 zug+I>qY!2H2>WevjWMZuxqXI*8VW;AxP?nomGl3oMmL^$Ta9Xb8gKc%`CNw(v)FV? zIcuE`1Sp2eD;;ye}^6$=X2fxVRW0kn+=w>Gioc0f%|Oz-lZ>Bm*VT(a1)DV$paIT=T|%_-7$ zb;nWZMu>xpL3rWt@<@J*~Y@feTlWoPHZtOJIU4+i94)0(I-H*O zz10oQ)Nzg8TEC+`w^~#Twpg@S`7zIxvhJ_9IbmXQ}4^>aOSmul){YwqvDN-d;&o zei^#37NC1I}Wy+H&eF1;|t??j^)K9jW;;ginG$Z}Iq(o|;(Ah;5!ygo_y2Ph(J zrz-(@Nej>FG~Y%pdB@X2d|82FNi%pJ@Dy~^PA+O@6yf=xgoSabDslGPPP{$1Lsu!9 zdR!7b53-iGSJr%Gl`M&BJeq~UCLGIdkc|m@v7@s*UrYViF+rzHqGzo;y-UsO$iQ3h zd0zMtMAKYi#DO|qe%^XFtdJQ{OSYNV!aE2vsqh(CXDg1iVo{PN+MB?MTC%|hO^E*Y z6@SgmD$5zi0@qL=?2v1Ud4b}uWvOwfuNVHnB$P_Gx>-s_HV4*ere~OLcZGm~TnC6H zWRVj*On)`>2scb+#*p}?m`09)y=pquG6P?AJT|)o9QBvn{d>=eKX`AFxg2f<3MZJV zwp9=eCuhnQydWNTBoi(D4Ltu$q_+a*ly-^x;+z8;S@)dF?5kQu!(p7x&B1K#V15ier3+!pDF z%jRZ!&JNg}-Lit$@=A7G=y}_@efm@>MSmfusa=|h80RblzDsJ9yibc3SMRfvNoO?P zIqs7nof9%5m!{3Mu(nc3hH4LFn!`gN*QA=aQ}dJ zeiR0gXR7IDI{i)lJQ)|wr_45u=~G4%#5F z|Kg}MJYN5XIg#^JZBA$;pmKjOCropGYEB${!<@L`#;(v!E`K0|u6BayxQ6m)9dDfo zj_#R*a$U$~BPmJfzUQ16q{n#ohDidyK=6;G>IX*`_RAdInRYkTB3ki+s<=n#i~4SQ z9$n0R8=p3Als~D$Yp*+Ig$%mi}#pB1r@8Y&ua%VeOJ+Rj2U$%G6>~#W?oq!RCxXJfU9zMdiC-LJzK90YpNq;9t zmkAl3fv@B!<9j*EP|MMAPHX9E$n%JuYF>8B38?25Q)iXo;<;tVk;|a46?-T{-9sDH zJ@juY_R!fkduTC{hFXpE35H@V0s;L%z|zUPB3a@H*KKUbw>vm+&>Z<9xxA$$4BECt z$cbK@eTaKU7yYiNu7p{E{-KLr{l_l4gzixCRg}K--<6a_@x%%xB}Y8fwM&eXvH2=N zwnoLSVD!RlC(QnsQu*@Ya^?|`{M)J6c^GP`uSf&b0Yff-+WphH{Ot>-D^y$l6(^9J zoqW!;V;XcBeJx(b&A=W3QPHxOtZ~)ii^oNvq})8aWiIEU3u_u3DZ|0SKKNxqAQq{v zNb|hlI4#jf6NMm zjPI*}4>OEfBTcnkVm2gsuhd76$B^s_%G01oS5cZ`z}C1Qi0Ux0qX5Ic3l9u_^!^3R zN1z4+V9L{8fsp5;ZW2am^-%%)`XeN{*-%>Fkf?~+XMg9}CcTPLaBi=J(z0*zq z!qq1+KO1_qhz`_<`1F#@ugb90rt}bfhpua~e*b5$rn~LK`4-6MEyI49o8P}={}ry_ zo2h?R9rpj0ONrhI3NQTXXD8$}l7}aQCD)I9fQhek8PhMjvsBOYj3zSmt2rY%ep8ms zZ#WV}ovM}Q<7IV!=Ten1L;+4XI#Ec?f~O~Axio!ikAhi7B#))ZTk_fOZIWSSY9Q!L z#2RM_uC--hs&J+;#~JIhzl$QyeW~|$kNYPak%gx#)S z3X*)19jLX>;hx~o=G&<@Pd_LF+PCfS=nlwEkSO~%6oh*1oTsDAIKb=1&??2;o zm1nz|oY^@&X%j|U(=^?9b@JsvYf$(owvvxr`Zj>_#>D6#JQ(5~8-3ap>~o1W`4FF# zLX7taV$&#Rag<}Y3U&0t?z#71M{W%+T5qpVWOTn$|7T}L$12Tlx9hWZ`;)9usFMvb zfA`Wk>)#!`kp6JVsk5=3JF|d`+miN;O`HvH1LRcOTy+PH*`=3L> zXSL$|x4QIV0_qOdWk)g2g;P*=&=Y)}ApgijwawJcX}7)n2PYDoxS*cs^_yXmA&}FD%6>A@3wP?7YA4{(E1c!l`}!kS||liDVlHHg^IWyf)vy zJP&=o?{@hlP?ib4P~|@EdH#8-Wn0&dG?ybiV^2jS<@{j~DRwO+}A>&-0KmK~8RTc|6Myg9FTkUS)Z#Mb9G zhIZLpg0?J)n7+9&NL~-I;d8P{8HQ6m$%L{qp$B;OqmEDu$UPwe@ek$APzM~i z;Ik&gg$B@0AJX4*knmNan#7vOTJrWF!85TXVg{4u`Z}(bFx!oYNxaYq^q%d#(cXDJ zGNEth5hm6iS|ML7^yD8eW{yh6MgxGk;Wy}!+Bbd@kwsM19VoBVHGVJp^oviX_^dn! zeLHyjo=2uUm6$kmZNnoQUizL~_1+pE1R&uTA)A-=E%=511-4&;UGF|eX>}EEmGi3x z#qaM${oYro0E|~abSf|~${d>2)D+^BckY!7?{+6UuZjRK^3{6|O&n3js?!s0xB5GN z^5E<*iiE91$pYR%$pU&T=QcdKR?MzH+Xc4G=bgLvqs-?q8RTF0$8y;*eg#^tXadW!=y*nb?A}r7cZ9c6iSp_AfhqivR9N z+W!j2|KEh8-r&9Tn9R-K#v@m$h*vwbBr!Q|>)(xF|Dvxd(c3}c!(V|14;_z=3V`yO z?racQk3?&i7k;Id(a%|g%UcW6Ug!1UpB`Qvop(>2)=!E1GP5}adZ;XSDl%LbUt@eI z{l&Ic;&*dG>QCo{CzDGrS0$AAk@oj{|FHMt)Q|Bi7eHQ{=1NRW;6&EB}`yM)d2?^@~$3{f(h>F zj{7}Lnh=z}y}N8-U3)EKljpiGe*Jb9`nMnHi#;HFF+?Ncl}Z3L@%&zEIjtl^Idg1g zJ_T^eO;<5>k;vzv7bpBP?3d%@eT(63-)$FoSJ3enQ?H*rkrGq&BKGAE5f5tA{1gdb zkby#}riSn=m9+EX5%iL|b;1IPxC&puQ;kenQTzY=Kg5kj;%c+Copr@CUlMq+>P^er zdevL8f84Y@G(GwKu4NnKL;_utABe9gdf4&5OH7YFw6HVz!6 z_abw?Huq?gZm@C)G2K#DSX%V?8RKE!z>q?nn~;HUw(%eodnC=+#1;D0?@bMUkNc^@ zM_-NJLNxTs96g$tkWW%s_TLcH5~wK?yb*v>$5&WiyxL5$z5QK^OPf|98&8k4PWvj&`#^~mZ^|Durh&X29M%E4Q8esiq^vR$MX z5Pg&xS7+$(iI{lL-vQcZyW8EA2s|&(n3tW@(ABFbM6N4yjl7c={@D?BGu~62WUejltRdJ;>)t znFFDh1y(-t5&#Td|6N|S-i^$Ww9m~%b;6EY9-%QrzF6(OE=vJ*)8iOKl4*`fb~7#; zD+!?-+ro@E8pNdcwQG{^2BaLXr;l|!q{Khmu#(^Xn~%w@no*JFJ9%(RxaI3dt$0;n z5q7q@eBB$y9E5U~G`({E`nQh{ru!ZbKPW;BC=i2^52@t&RjV*Rre+x`en(^V(Mavp z+En4`|A^-0=zYrPI2ecQTVr$eZyxS3-F_{ltpz60MrHR~3syP&_34*eM(CDXjoY@u z;efUFZNB?#)++y2AdBsrL`FaV?`>_{do&bFe}x1bS{7vISoBWP5&#tRf%0+uTFjmS zZ*VhNNMnbk5Q8UZoaw8hh|!q-@+R)k33R;uz-p^r-`d!i{tQL6M}}2U@8HKSs-Uvb zn3xUr@%C%=T&ql&=S-W!u&{N0y~PKhNfxbTWh|%6X5(#JoSe1x#kkSx%W-#ZY+^=1 zH*-h+v-E`AJF6t+kyBR-+b=)+Um*|i-k*^lfx`Y-YNKD|ZqBGEJfU5c<`n5sV32-m z;ArR0BaM-mXxvr8Erbz!Rc7?v@PAWL3QL1kPdup!!JfhnKnJ7t?9n%9E92<8$VuF;y+Oi_5!>$`Huvdz$S#Sn2g9I1&UeqO?GGJ>XNikR&$~qSMA~ zDQ~{oD}Y!s#KO;k5+Yg)`RlXiuya8#nmtd0KspuUz1+T0V)o7Z{DOx1-TAx$-C|Gn zQovRcIWRi`c4GZ}M>5^q*pviY9)3)()P;osx#b``oJsJYNYj91C6q=WH`FxKCh~o? z(tAso4+o9i{Hb+-G2t9TCNk5htnu}7adawE;>K4Mp4^n3rA?D|7Orx=?&C# zOze>i6N-unQY;L8R=dzGFrdoymnEH=i?%MPmvp+RZi8aDA8ShO;3H+2I=5cR(j0;( zrq(XZTn}C$>iFAZATle>!{+u-JSaNBeWjYIrYBbZFgCTDU(iQB_dY!wrW z=RY0rN*NmDgo0%3vvz~Gyc3Lo3fzM%F6Xm_7mH?>-0Ahkm5tgYFv{YxH?AXKu6uW7 zg}o+3ml)u_X9rL}%q~km?6PY$}-UDET^92yfG#inH8VSy|)xcr=4hk6_+`x`lZs$+2Dh^uxiGWH@~dn;4JXx z_a=*vCPBD9>u`RnPIyf04GPS%Pj4`)w{wMDP$46x46RD?sca z*o8VrPdriAv2#^I#&G0*>xxi<3B^}5IA3?A(zzx9%Tm*|;bFK%LB0#Ara8%{n|RTT ztI+anW1?P+&0S+@k}Dl7Dg}itTs@O)9$_k5s@9>;AmG7fzf&F3zQA^~r^Y6welqjr zMWQm;^aOO4J46@^cV$aTXw@EGrWK?rX0WN=X{sFucB3Nol9NH?bH51AT8H4-p3|;~lJ>I=I@7ZyTS}YJMOx-raL&8gOq9^2+aFb^Y%5P{lTx_7AY%e_PqV!WAmjh-Mh! zVY$?>4hM8e-KR&lIT~lfH`>2z9rdxL;+`%P0jf3t8uecCRS$r!0EX4EevKT3e*=u`G{ zror~AWeEmD?2e1E=k@E(BO$EyPmGP^vz|2PSW5YPQfe_DNDG>J5&mN}x z2!cFg(pof{*ivR66s^TfM47i0B!kdBsww&&49eE4hqWY<1=OM&YX_T5@Qr7j6|$W< zE(VsQl!=~d`)y4k1bwt78%t{>`tDoDWaa}Xa)4^F5v}mK@MsQBRL;Wd^Abp~yn(@T zR`cj*eYzSYvB_Zveefop*p1%ME))Qi|sEXK{4NWXFcohg{{bTddtmkJ9+ z-_U}%SrN6!ZD4Dm$n$uOW{u>D!~@B5aq*zpp+*fpbz6_rC7Hfyx}sRXn;V*_Q4^@y zA$V+4Bq`E1%vB&dUjniNATHsAW&^giqGpuGgJvz_x$v(`(8TUYTiewDJ3#V=GWYD! zN7;fJM&+NY&ardKDceW8yT=P#{j=6$b`+;=RX-}iy9fNmvDCo&^3<@%zd=eACQLT1 ze@Ey@XjRAY-`%wN9d5WRK1e-w5vhYcWtV22TY1_T{dGNzC+e?bVjP*( zv1Vk?h!n%Y+|0Oo&6d$FsNf`wf@L&#)K8#ph_Gx#V9(W$ybfyQOhs?>a{#6!Sa(f zDi>a0hVuj8iS|^5hEL>DF({v17+u6ig?oWJj^R`v4yz8hVtL+*=>~LFuSOa z6C`YaH!$v&^2IcrMP2WLTKIOjGaRoJm!-I%qzAo#OEfyISyO{jx4?0U`Rbk6rkA&N zo;Ebaxt>DgahSYTQ8hG;M9WvsCf0s}sGyw}PgMitp^tRK+bSA27bx3ZnWCw`Ce9{q zbx<1-S-n2#Tz%^HrxoB}!qVBBZfG got = module().score(text, '73') + +opt_prime/tests/test_heldout_quality.py:25: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + + def module(): +> assert path.exists(), 'held-out evaluator is not implemented' +E AssertionError: held-out evaluator is not implemented +E assert False +E + where False = exists() +E + where exists = PosixPath('/home/gspark/workspace/aicomp-heldout-quality-263/opt_prime/examples/pd/http_matrix/heldout_quality.py').exists + +opt_prime/tests/test_heldout_quality.py:9: AssertionError +______________ test_score[{"answer":"73","note":"ok"}-True-False] ______________ + +text = '{"answer":"73","note":"ok"}', correct = True, formatted = False + + @pytest.mark.parametrize('text,correct,formatted', [ + ('{"answer":"73"}', True, True), + ('{"answer":"73","note":"ok"}', True, False), + ('{"answer":73}', False, False), + ('The answer is 73', False, False), + ('{"answer":"173"}', False, True), + ('{"answer":"wrong","answer":"73"}', False, False), + ('{"answer":"73"} trailing', False, False), + ]) + def test_score(text, correct, formatted): +> got = module().score(text, '73') + +opt_prime/tests/test_heldout_quality.py:25: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + + def module(): +> assert path.exists(), 'held-out evaluator is not implemented' +E AssertionError: held-out evaluator is not implemented +E assert False +E + where False = exists() +E + where exists = PosixPath('/home/gspark/workspace/aicomp-heldout-quality-263/opt_prime/examples/pd/http_matrix/heldout_quality.py').exists + +opt_prime/tests/test_heldout_quality.py:9: AssertionError +____________________ test_score[{"answer":73}-False-False] _____________________ + +text = '{"answer":73}', correct = False, formatted = False + + @pytest.mark.parametrize('text,correct,formatted', [ + ('{"answer":"73"}', True, True), + ('{"answer":"73","note":"ok"}', True, False), + ('{"answer":73}', False, False), + ('The answer is 73', False, False), + ('{"answer":"173"}', False, True), + ('{"answer":"wrong","answer":"73"}', False, False), + ('{"answer":"73"} trailing', False, False), + ]) + def test_score(text, correct, formatted): +> got = module().score(text, '73') + +opt_prime/tests/test_heldout_quality.py:25: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + + def module(): +> assert path.exists(), 'held-out evaluator is not implemented' +E AssertionError: held-out evaluator is not implemented +E assert False +E + where False = exists() +E + where exists = PosixPath('/home/gspark/workspace/aicomp-heldout-quality-263/opt_prime/examples/pd/http_matrix/heldout_quality.py').exists + +opt_prime/tests/test_heldout_quality.py:9: AssertionError +___________________ test_score[The answer is 73-False-False] ___________________ + +text = 'The answer is 73', correct = False, formatted = False + + @pytest.mark.parametrize('text,correct,formatted', [ + ('{"answer":"73"}', True, True), + ('{"answer":"73","note":"ok"}', True, False), + ('{"answer":73}', False, False), + ('The answer is 73', False, False), + ('{"answer":"173"}', False, True), + ('{"answer":"wrong","answer":"73"}', False, False), + ('{"answer":"73"} trailing', False, False), + ]) + def test_score(text, correct, formatted): +> got = module().score(text, '73') + +opt_prime/tests/test_heldout_quality.py:25: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + + def module(): +> assert path.exists(), 'held-out evaluator is not implemented' +E AssertionError: held-out evaluator is not implemented +E assert False +E + where False = exists() +E + where exists = PosixPath('/home/gspark/workspace/aicomp-heldout-quality-263/opt_prime/examples/pd/http_matrix/heldout_quality.py').exists + +opt_prime/tests/test_heldout_quality.py:9: AssertionError +___________________ test_score[{"answer":"173"}-False-True] ____________________ + +text = '{"answer":"173"}', correct = False, formatted = True + + @pytest.mark.parametrize('text,correct,formatted', [ + ('{"answer":"73"}', True, True), + ('{"answer":"73","note":"ok"}', True, False), + ('{"answer":73}', False, False), + ('The answer is 73', False, False), + ('{"answer":"173"}', False, True), + ('{"answer":"wrong","answer":"73"}', False, False), + ('{"answer":"73"} trailing', False, False), + ]) + def test_score(text, correct, formatted): +> got = module().score(text, '73') + +opt_prime/tests/test_heldout_quality.py:25: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + + def module(): +> assert path.exists(), 'held-out evaluator is not implemented' +E AssertionError: held-out evaluator is not implemented +E assert False +E + where False = exists() +E + where exists = PosixPath('/home/gspark/workspace/aicomp-heldout-quality-263/opt_prime/examples/pd/http_matrix/heldout_quality.py').exists + +opt_prime/tests/test_heldout_quality.py:9: AssertionError +___________ test_score[{"answer":"wrong","answer":"73"}-False-False] ___________ + +text = '{"answer":"wrong","answer":"73"}', correct = False, formatted = False + + @pytest.mark.parametrize('text,correct,formatted', [ + ('{"answer":"73"}', True, True), + ('{"answer":"73","note":"ok"}', True, False), + ('{"answer":73}', False, False), + ('The answer is 73', False, False), + ('{"answer":"173"}', False, True), + ('{"answer":"wrong","answer":"73"}', False, False), + ('{"answer":"73"} trailing', False, False), + ]) + def test_score(text, correct, formatted): +> got = module().score(text, '73') + +opt_prime/tests/test_heldout_quality.py:25: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + + def module(): +> assert path.exists(), 'held-out evaluator is not implemented' +E AssertionError: held-out evaluator is not implemented +E assert False +E + where False = exists() +E + where exists = PosixPath('/home/gspark/workspace/aicomp-heldout-quality-263/opt_prime/examples/pd/http_matrix/heldout_quality.py').exists + +opt_prime/tests/test_heldout_quality.py:9: AssertionError +_______________ test_score[{"answer":"73"} trailing-False-False] _______________ + +text = '{"answer":"73"} trailing', correct = False, formatted = False + + @pytest.mark.parametrize('text,correct,formatted', [ + ('{"answer":"73"}', True, True), + ('{"answer":"73","note":"ok"}', True, False), + ('{"answer":73}', False, False), + ('The answer is 73', False, False), + ('{"answer":"173"}', False, True), + ('{"answer":"wrong","answer":"73"}', False, False), + ('{"answer":"73"} trailing', False, False), + ]) + def test_score(text, correct, formatted): +> got = module().score(text, '73') + +opt_prime/tests/test_heldout_quality.py:25: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + + def module(): +> assert path.exists(), 'held-out evaluator is not implemented' +E AssertionError: held-out evaluator is not implemented +E assert False +E + where False = exists() +E + where exists = PosixPath('/home/gspark/workspace/aicomp-heldout-quality-263/opt_prime/examples/pd/http_matrix/heldout_quality.py').exists + +opt_prime/tests/test_heldout_quality.py:9: AssertionError +__________ test_paired_regressions_are_not_hidden_by_equal_aggregate ___________ + + def test_paired_regressions_are_not_hidden_by_equal_aggregate(): +> m = module() + +opt_prime/tests/test_heldout_quality.py:29: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + + def module(): +> assert path.exists(), 'held-out evaluator is not implemented' +E AssertionError: held-out evaluator is not implemented +E assert False +E + where False = exists() +E + where exists = PosixPath('/home/gspark/workspace/aicomp-heldout-quality-263/opt_prime/examples/pd/http_matrix/heldout_quality.py').exists + +opt_prime/tests/test_heldout_quality.py:9: AssertionError +___________ test_context_builder_keeps_retrieval_fact_inside_budget ____________ + + def test_context_builder_keeps_retrieval_fact_inside_budget(): + class CharacterTokenizer: + def apply_chat_template(self, messages, **kwargs): + return list(messages[0]['content'].encode()) +> m = module() + +opt_prime/tests/test_heldout_quality.py:46: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + + def module(): +> assert path.exists(), 'held-out evaluator is not implemented' +E AssertionError: held-out evaluator is not implemented +E assert False +E + where False = exists() +E + where exists = PosixPath('/home/gspark/workspace/aicomp-heldout-quality-263/opt_prime/examples/pd/http_matrix/heldout_quality.py').exists + +opt_prime/tests/test_heldout_quality.py:9: AssertionError +=========================== short test summary info ============================ +FAILED opt_prime/tests/test_heldout_quality.py::test_score[{"answer":"73"}-True-True] +FAILED opt_prime/tests/test_heldout_quality.py::test_score[{"answer":"73","note":"ok"}-True-False] +FAILED opt_prime/tests/test_heldout_quality.py::test_score[{"answer":73}-False-False] +FAILED opt_prime/tests/test_heldout_quality.py::test_score[The answer is 73-False-False] +FAILED opt_prime/tests/test_heldout_quality.py::test_score[{"answer":"173"}-False-True] +FAILED opt_prime/tests/test_heldout_quality.py::test_score[{"answer":"wrong","answer":"73"}-False-False] +FAILED opt_prime/tests/test_heldout_quality.py::test_score[{"answer":"73"} trailing-False-False] +FAILED opt_prime/tests/test_heldout_quality.py::test_paired_regressions_are_not_hidden_by_equal_aggregate +FAILED opt_prime/tests/test_heldout_quality.py::test_context_builder_keeps_retrieval_fact_inside_budget +9 failed in 0.04s diff --git a/opt_prime/examples/pd/http_matrix/experiments/2026-10-24-heldout-quality263.md b/opt_prime/examples/pd/http_matrix/experiments/2026-10-24-heldout-quality263.md new file mode 100644 index 00000000..08380023 --- /dev/null +++ b/opt_prime/examples/pd/http_matrix/experiments/2026-10-24-heldout-quality263.md @@ -0,0 +1,42 @@ +# Held-out structured answers across four model families + +- **Date**: 2026-10-24 +- **Task**: Structured answer compatibility (aicomp #263) +- **Host / device**: GPU and RNGD+ATOM roles; hardware results pending +- **Commit**: evaluator based on f9ce6e8, exp/263-heldout-quality (uncommitted); production runtime 32ce5d5 + +## Question + +Do token differences change structured-answer correctness across native GPU, RNGD→GPU, native ATOM, and RNGD→ATOM? Models: Qwen3-0.6B, Qwen3-4B, KONI-8B, Gemma3-1B. Qwen2.5 excluded at user request. + +## Setup + +Preregistered before new inference: seed 26320261024; three tasks (addition, passkey retrieval, Korean table lookup), four variants, short/~2048/~8192 input tokens = 36 cases per model. Same serialized chat-template input IDs for every route, thinking disabled, greedy, normal EOS, output cap 128. Retrieval fact is in the middle of deterministic filler. Context sizes include the chat template. No answer-conditioned prompt edits. + +Exact string answer in a parsed JSON object is scored separately from the one-key JSON format requirement. Duplicate JSON keys, non-string answers, trailing prose, and malformed JSON fail. This is structured-answer adherence, not a universal semantic scorer. Retain complete responses and token IDs; compare paired correct→wrong, wrong→correct, and exact tokens independently. Any transport or inference failure is retained and stops the route; do not silently exclude it. Fixture hashes identify paired inputs. + +## Command + +```bash +# In the isolated aicomp worktree: +python -m pytest opt_prime/tests/test_heldout_quality.py -q +# Create each fixture file once, before inference: +python opt_prime/examples/pd/http_matrix/heldout_quality.py fixtures --model "$MODEL" --output "$FIXTURE" +python opt_prime/examples/pd/http_matrix/heldout_quality.py run --fixtures "$FIXTURE" --endpoint "$ENDPOINT" --route "$ROUTE" --output "$RESULT" +``` + +Local Python: /home/gspark/venvs/pt213/bin/python. Full-suite environment: OMP_NUM_THREADS=4 MKL_NUM_THREADS=4 TORCH_DEVICE_BACKEND_AUTOLOAD=0. + +## Results + +- Test-first: 9 failures for missing evaluator, then 9 passed in 0.43 seconds. +- HTTP boundary test caught missing request ID: 1 failed/9 passed before fix, then 10 passed. +- Full suite collected before that last test: 533 passed, 15 skipped, 253 subtests in 160.17 s; 13 warnings from existing pinned-memory deprecations and unfused flex attention. +- All 144 input cases frozen and SHA256 manifest saved before inference. GPU paired runs started; no new model accuracy or performance claim. +- Existing Qwen3-4B/KONI exact-token failures remain open (#256/#261). + +Raw logs: /tmp/quality263-red.log, /tmp/quality263-full.log; fixtures/results under analysis/serving-matrix-201/quality263-*. + +## Conclusion + +This small synthetic set can expose paired regressions but cannot establish general model quality or statistical equivalence. The compatibility preference question remains unanswered; exact token failures are not waived. diff --git a/opt_prime/examples/pd/http_matrix/heldout_quality.py b/opt_prime/examples/pd/http_matrix/heldout_quality.py new file mode 100644 index 00000000..af1ab56e --- /dev/null +++ b/opt_prime/examples/pd/http_matrix/heldout_quality.py @@ -0,0 +1,133 @@ +"""Preregistered synthetic structured-answer checks; not a general quality benchmark.""" +import argparse +import hashlib +import json +import random +import time +import urllib.request +from pathlib import Path + + +def score(text, expected): + def unique(pairs): + result = {} + for key, value in pairs: + if key in result: + raise ValueError('duplicate JSON key') + result[key] = value + return result + try: + value = json.loads(text, object_pairs_hook=unique) + except (ValueError, TypeError): + value = None + valid = isinstance(value, dict) and isinstance(value.get('answer'), str) + return dict(answer_correct=bool(valid and value['answer'] == expected), + format_correct=bool(valid and set(value) == {'answer'})) + + +def compare(baseline, candidate): + left = {x['case']: x for x in baseline} + right = {x['case']: x for x in candidate} + if len(left) != len(baseline) or len(right) != len(candidate) or left.keys() != right.keys(): + raise ValueError('duplicate or missing paired cases') + result = dict(cases=len(left), correct_to_wrong=0, wrong_to_correct=0, token_equal=0) + for key, a in left.items(): + b = right[key] + if a['input_sha256'] != b['input_sha256']: + raise ValueError('paired input IDs differ') + result['correct_to_wrong'] += bool(a['answer_correct'] and not b['answer_correct']) + result['wrong_to_correct'] += bool(not a['answer_correct'] and b['answer_correct']) + result['token_equal'] += a['tokens'] == b['tokens'] + return result + + +def fixtures(tokenizer): + rng = random.Random(26320261024) + tasks = [] + for variant in range(4): + a, b = rng.randrange(21, 80), rng.randrange(11, 60) + tasks.append(('arithmetic', variant, '', f'Compute {a} + {b}.', str(a + b))) + secret = ''.join(rng.choice('ABCDEFGHJKLMNPQRSTUVWXYZ23456789') for _ in range(8)) + tasks.append(('passkey', variant, f'The access key for archive {variant} is {secret}.', + f'What is the access key for archive {variant}?', secret)) + city = ['춘천', '목포', '전주', '속초'][variant] + tasks.append(('korean_table', variant, + f'배송표: 주문 가{variant}=대전; 주문 나{variant}={city}; 주문 다{variant}=부산.', + f'배송표에서 주문 나{variant}의 도착 도시를 찾으세요.', city)) + filler = 'Background: shelves hold ordinary books; this note contains no task answer.\n' + result = [] + for target in [0, 2048, 8192]: + for task, variant, fact, question, expected in tasks: + def render(repeats): + prompt = (filler * (repeats // 2) + fact + '\n' + filler * (repeats - repeats // 2) + + '\n' + question + '\nReturn only a JSON object with one key "answer" and a string value. No explanation.') + ids = tokenizer.apply_chat_template([{'role': 'user', 'content': prompt}], + tokenize=True, return_dict=False, add_generation_prompt=True, enable_thinking=False) + return prompt, [int(x) for x in ids] + low, high = 0, target + if target and len(render(0)[1]) > target: + raise ValueError('base prompt exceeds context target') + while low < high: + mid = (low + high + 1) // 2 + if len(render(mid)[1]) <= target: + low = mid + else: + high = mid - 1 + prompt, ids = render(low) + digest = hashlib.sha256(json.dumps(ids, separators=(',', ':')).encode()).hexdigest() + result.append(dict(case=f'{task}-{variant}-{target}', task=task, target_tokens=target, + expected=expected, prompt=prompt, ids=ids, input_sha256=digest)) + return result + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + sub = parser.add_subparsers(dest='command', required=True) + export = sub.add_parser('fixtures') + export.add_argument('--model', required=True) + export.add_argument('--output', type=Path, required=True) + run = sub.add_parser('run') + run.add_argument('--fixtures', type=Path, required=True) + run.add_argument('--endpoint', required=True) + run.add_argument('--route', required=True) + run.add_argument('--output', type=Path, required=True) + args = parser.parse_args() + if args.command == 'fixtures': + from transformers import AutoTokenizer + tokenizer = AutoTokenizer.from_pretrained(args.model, local_files_only=True) + data = dict(model=args.model, seed=26320261024, max_tokens=128, + thinking=False, cases=fixtures(tokenizer)) + args.output.write_text(json.dumps(data, ensure_ascii=False, indent=2) + '\n') + return + data = json.loads(args.fixtures.read_text()) + # Exclusive creation prevents silently replacing a prior run after seeing answers. + with args.output.open('x') as output: + for case in data['cases']: + body = dict(model=data['model'], prompt=case['ids'], temperature=0, + max_tokens=data['max_tokens'], ignore_eos=False, stream=False) + request = urllib.request.Request(args.endpoint.rstrip('/') + '/v1/completions', + data=json.dumps(body).encode(), headers={'Content-Type': 'application/json', + 'x-request-id': 'quality263-' + case['case']}) + start = time.perf_counter() + row = dict(case=case['case'], route=args.route, input_sha256=case['input_sha256'], + input_tokens=len(case['ids']), expected=case['expected']) + try: + with urllib.request.urlopen(request, timeout=600) as response: + payload = json.load(response) + choice = payload['choices'][0] + row.update(text=choice['text'], tokens=choice['token_ids'], + finish_reason=choice['finish_reason'], response=payload, + **score(choice['text'], case['expected'])) + except Exception as error: + row.update(error=repr(error), answer_correct=False, format_correct=False) + output.write(json.dumps(row, ensure_ascii=False) + '\n') + output.flush() + raise + row['elapsed_seconds'] = time.perf_counter() - start + output.write(json.dumps(row, ensure_ascii=False) + '\n') + output.flush() + print(case['case'], row['answer_correct'], row['format_correct'], flush=True) + + +if __name__ == '__main__': + main() diff --git a/opt_prime/tests/test_heldout_quality.py b/opt_prime/tests/test_heldout_quality.py new file mode 100644 index 00000000..81a74343 --- /dev/null +++ b/opt_prime/tests/test_heldout_quality.py @@ -0,0 +1,92 @@ +"""Prevent answer substrings, duplicate keys, or missing pairs from inflating scores.""" +import importlib.util +from pathlib import Path +import pytest + +path = Path(__file__).parents[1] / 'examples/pd/http_matrix/heldout_quality.py' + +def module(): + assert path.exists(), 'held-out evaluator is not implemented' + spec = importlib.util.spec_from_file_location('heldout_quality', path) + value = importlib.util.module_from_spec(spec) + spec.loader.exec_module(value) + return value + +@pytest.mark.parametrize('text,correct,formatted', [ + ('{"answer":"73"}', True, True), + ('{"answer":"73","note":"ok"}', True, False), + ('{"answer":73}', False, False), + ('The answer is 73', False, False), + ('{"answer":"173"}', False, True), + ('{"answer":"wrong","answer":"73"}', False, False), + ('{"answer":"73"} trailing', False, False), +]) +def test_score(text, correct, formatted): + got = module().score(text, '73') + assert (got['answer_correct'], got['format_correct']) == (correct, formatted) + +def test_paired_regressions_are_not_hidden_by_equal_aggregate(): + m = module() + base = [{'case': 'a', 'input_sha256': 'x', 'answer_correct': True, 'tokens': [1]}, + {'case': 'b', 'input_sha256': 'y', 'answer_correct': False, 'tokens': [2]}] + other = [{'case': 'a', 'input_sha256': 'x', 'answer_correct': False, 'tokens': [3]}, + {'case': 'b', 'input_sha256': 'y', 'answer_correct': True, 'tokens': [2]}] + got = m.compare(base, other) + assert got == {'cases': 2, 'correct_to_wrong': 1, 'wrong_to_correct': 1, 'token_equal': 1} + with pytest.raises(ValueError): + m.compare(base, other[:1]) + other[0]['input_sha256'] = 'changed' + with pytest.raises(ValueError): + m.compare(base, other) + +def test_context_builder_keeps_retrieval_fact_inside_budget(): + class CharacterTokenizer: + def apply_chat_template(self, messages, **kwargs): + return list(messages[0]['content'].encode()) + m = module() + cases = m.fixtures(CharacterTokenizer()) + assert len(cases) == len({x['case'] for x in cases}) == 36 + for case in cases: + assert len(case['ids']) <= (case['target_tokens'] or 2048) + if case['task'] != 'arithmetic': + assert case['expected'] in case['prompt'] + if case['target_tokens']: + assert len(case['ids']) > case['target_tokens'] - 150 + +def test_http_runner_supplies_required_request_id_and_frozen_tokens(tmp_path, monkeypatch): + import json + import sys + import threading + from http.server import BaseHTTPRequestHandler, HTTPServer + seen = [] + class Handler(BaseHTTPRequestHandler): + def do_POST(self): + body = json.loads(self.rfile.read(int(self.headers['Content-Length']))) + seen.append((self.headers.get('x-request-id'), body)) + self.send_response(200) + self.end_headers() + self.wfile.write(json.dumps({'choices': [{'text': '{"answer":"73"}', + 'token_ids': [9], 'finish_reason': 'stop'}]}).encode()) + def log_message(self, *args): + pass + server = HTTPServer(('127.0.0.1', 0), Handler) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + fixture = tmp_path / 'input.json' + output = tmp_path / 'output.jsonl' + fixture.write_text(json.dumps({'model':'model', 'max_tokens':128, + 'cases':[{'case':'a','ids':[4,5], 'input_sha256':'hash','expected':'73'}]})) + monkeypatch.setattr(sys, 'argv', ['heldout_quality', 'run', '--fixtures',str(fixture), + '--endpoint',f'http://127.0.0.1:{server.server_port}', '--route','test', '--output',str(output)]) + try: + module().main() + finally: + server.shutdown() + thread.join() + server.server_close() + assert seen[0][0], 'stage HTTP API requires a nonempty request ID' + assert seen[0][1]['prompt'] == [4,5] + assert seen[0][1]['ignore_eos'] is False + assert json.loads(output.read_text())['answer_correct'] is True + with pytest.raises(FileExistsError): + module().main() From b3b4a5ad3311d72dddcee09b39921ec0feefa8a3 Mon Sep 17 00:00:00 2001 From: gspark Date: Sat, 24 Oct 2026 05:29:17 +0900 Subject: [PATCH 2/7] test: record Qwen held-out GPU pairs and ATOM execution setup --- opt_prime/examples/pd/http_matrix/STATUS.md | 2 + .../quality263-atom-source-sha256.txt | 8 ++ .../quality263-qwen06b-gpu-summary.json | 10 ++ .../quality263-qwen06b-native-gpu-run.json | 43 +++++++ .../quality263-qwen06b-native-gpu-server.log | 3 + .../quality263-qwen06b-native-gpu.jsonl | 36 ++++++ .../quality263-qwen06b-rngd-gpu-run.json | 47 ++++++++ .../quality263-qwen06b-rngd-gpu-server.log | 5 + .../quality263-qwen06b-rngd-gpu.jsonl | 36 ++++++ .../quality263-qwen4b-gpu-summary.json | 10 ++ .../quality263-qwen4b-native-gpu-run.json | 43 +++++++ .../quality263-qwen4b-native-gpu-server.log | 3 + .../quality263-qwen4b-native-gpu.jsonl | 36 ++++++ .../quality263-qwen4b-rngd-gpu-run.json | 47 ++++++++ .../quality263-qwen4b-rngd-gpu-server.log | 5 + .../quality263-qwen4b-rngd-gpu.jsonl | 36 ++++++ .../evidence263/quality263_atom.py | 111 ++++++++++++++++++ .../evidence263/run-quality263-atom.sh | 10 ++ .../evidence263/run_quality263_gpu.py | 79 +++++++++++++ .../2026-10-24-heldout-quality263.md | 19 ++- 20 files changed, 588 insertions(+), 1 deletion(-) create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-atom-source-sha256.txt create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen06b-gpu-summary.json create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen06b-native-gpu-run.json create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen06b-native-gpu-server.log create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen06b-native-gpu.jsonl create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen06b-rngd-gpu-run.json create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen06b-rngd-gpu-server.log create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen06b-rngd-gpu.jsonl create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen4b-gpu-summary.json create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen4b-native-gpu-run.json create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen4b-native-gpu-server.log create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen4b-native-gpu.jsonl create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen4b-rngd-gpu-run.json create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen4b-rngd-gpu-server.log create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen4b-rngd-gpu.jsonl create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263_atom.py create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/run-quality263-atom.sh create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/run_quality263_gpu.py diff --git a/opt_prime/examples/pd/http_matrix/STATUS.md b/opt_prime/examples/pd/http_matrix/STATUS.md index 67c05135..08ccbc00 100644 --- a/opt_prime/examples/pd/http_matrix/STATUS.md +++ b/opt_prime/examples/pd/http_matrix/STATUS.md @@ -51,3 +51,5 @@ Allfour live GPU isolation runs pass: active512-token decode survives other-requ #263 held-out suite: Qwen3-0.6B/Qwen3-4B/KONI-8B/Gemma3-1B inputs frozen (36/model); scorer distinguishes parsed answer, JSON format and exact token parity. Tests9passed after9expected failures. Hardware results pending; #256/#261 remain open. #263 evaluator verification:10focused passed; full prior collection533passed/15skipped/253subtests. Frozen fixtures hashed before inference; GPU route queue started, hardware results pending. + +#263 first GPU pairs complete: Qwen0.6 native32/36 versus RNGD34/36, no paired regression,34/36 token equality; Qwen4 both36/36, alltokens equal. KONI/Gemma and ATOM queues still running; old long-output failures are retained. diff --git a/opt_prime/examples/pd/http_matrix/evidence263/quality263-atom-source-sha256.txt b/opt_prime/examples/pd/http_matrix/evidence263/quality263-atom-source-sha256.txt new file mode 100644 index 00000000..49d3e1b7 --- /dev/null +++ b/opt_prime/examples/pd/http_matrix/evidence263/quality263-atom-source-sha256.txt @@ -0,0 +1,8 @@ +c6674a711e3c071bd71a99fe2f155922b9b884340d16c7d25a7b9fdd8ec5b7e8 atom_batching253.py +dbccc7a1d45671e46ea1ba413b11e4e4820ffc65e17566e0ad8b398e2fb37b95 pd_batching253.py +81fa8c78648064c38a8146f52f5207553cf91fb5de8d9048f2de3340f4a8b316 batched253.py +849165a74a2046f7bc201d346201bf1fa98ef4d7ec0fe4bd71498614d5ec47fd llmd253.py +7fe175b5c13fb8a3d913d5f42af906e7916de2d5cb1cba64905e5e2cea8ca935 worker_python253.sh +f497a2c9adeabdd46a0fe2a62553dd6c3e9be632c09b82df3eef441557af0639 quality263_atom.py +b4ab0f8c7810006bcbb6fb239a5d9f56d8a2ea9624d7174ec33c7344a57f4624 heldout_quality263.py +587908a00c72826df0cb0b4175e1b0e219962b5a42a72200d0e42c2b9b89f1f0 run-quality263-atom.sh diff --git a/opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen06b-gpu-summary.json b/opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen06b-gpu-summary.json new file mode 100644 index 00000000..3aff3263 --- /dev/null +++ b/opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen06b-gpu-summary.json @@ -0,0 +1,10 @@ +{ + "cases": 36, + "correct_to_wrong": 0, + "wrong_to_correct": 2, + "token_equal": 34, + "answer_counts": [ + 32, + 34 + ] +} diff --git a/opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen06b-native-gpu-run.json b/opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen06b-native-gpu-run.json new file mode 100644 index 00000000..c25ee341 --- /dev/null +++ b/opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen06b-native-gpu-run.json @@ -0,0 +1,43 @@ +{ + "label": "qwen06b", + "route": "native-gpu", + "argv": [ + "/home/gspark/venvs/pt213/bin/python", + "-c", + "import sys,importlib.util,runpy\ns=importlib.util.spec_from_file_location('opt_prime.accel.serve.llmd','/tmp/llmd240.py')\nm=importlib.util.module_from_spec(s);sys.modules[s.name]=m;s.loader.exec_module(m)\nrunpy.run_path('/home/gspark/workspace/aicomp-atom-prefetch-253/opt_prime/examples/pd/serve_batched.py',run_name='__main__')\n", + "--model", + "/home/gspark/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/c1899de289a04d12100db370d81485cdf75e47ca", + "--device", + "cuda:0", + "--dtype", + "bfloat16", + "--slots", + "4", + "--max-context", + "9216", + "--max-output", + "128", + "--kv", + "paged", + "--page-size", + "64", + "--chunk", + "256", + "--port", + "60413" + ], + "env": { + "OMP_NUM_THREADS": "4", + "MKL_NUM_THREADS": "4", + "TORCH_DEVICE_BACKEND_AUTOLOAD": "0", + "HF_HUB_OFFLINE": "1", + "TRANSFORMERS_OFFLINE": "1", + "PYTHONPATH": "/tmp/rngd-gpu201-src:/home/gspark/workspace/aicomp-live-pd-234/opt_prime/examples/pd/serving_matrix:/home/gspark/workspace/aicomp-dense-hybrid-258/opt_prime" + }, + "runtime_commit": "32ce5d5", + "stage": "complete", + "cases": 36, + "answer_correct": 32, + "format_correct": 36, + "server_exit": -15 +} diff --git a/opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen06b-native-gpu-server.log b/opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen06b-native-gpu-server.log new file mode 100644 index 00000000..0a0da88c --- /dev/null +++ b/opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen06b-native-gpu-server.log @@ -0,0 +1,3 @@ + +Loading weights: 0%| | 0/311 [00:00deadline: raise TimeoutError('owner startup') + time.sleep(.01) + result['stage']='running';save() + subprocess.run([sys.executable,'heldout_quality263.py','run','--fixtures',a.fixture, + '--endpoint',f'http://127.0.0.1:{server.server_port}','--route',a.route,'--output',a.output],check=True) + rows=[json.loads(line) for line in Path(a.output).read_text().splitlines()] + assert len(rows)==36 and all('error' not in row for row in rows) + deadline=time.monotonic()+30 + while not b.idle and time.monotonic()deadline: + raise TimeoutError('readiness timeout') + time.sleep(2) + subprocess.run([sys.executable,script,'run','--fixtures',str(fixture), + '--endpoint',endpoint,'--route',route,'--output',str(output)],env=env,check=True) + rows=[json.loads(line) for line in output.read_text().splitlines()] + assert len(rows)==36 and all('error' not in row for row in rows) + metadata.update(stage='complete',cases=36,answer_correct=sum(x['answer_correct'] for x in rows),format_correct=sum(x['format_correct'] for x in rows)) + except BaseException as error: + metadata.update(stage='failed',error=repr(error)) + raise + finally: + if proc.poll() is None: + os.killpg(proc.pid,signal.SIGTERM) + try: + proc.wait(timeout=30) + except subprocess.TimeoutExpired: + os.killpg(proc.pid,signal.SIGKILL) + proc.wait() + metadata['server_exit']=proc.returncode + meta.write_text(json.dumps(metadata,indent=2)+'\n') + print('END',label,route,metadata['answer_correct'],metadata['format_correct'],flush=True) + time.sleep(3) diff --git a/opt_prime/examples/pd/http_matrix/experiments/2026-10-24-heldout-quality263.md b/opt_prime/examples/pd/http_matrix/experiments/2026-10-24-heldout-quality263.md index 08380023..ce33236b 100644 --- a/opt_prime/examples/pd/http_matrix/experiments/2026-10-24-heldout-quality263.md +++ b/opt_prime/examples/pd/http_matrix/experiments/2026-10-24-heldout-quality263.md @@ -3,7 +3,7 @@ - **Date**: 2026-10-24 - **Task**: Structured answer compatibility (aicomp #263) - **Host / device**: GPU and RNGD+ATOM roles; hardware results pending -- **Commit**: evaluator based on f9ce6e8, exp/263-heldout-quality (uncommitted); production runtime 32ce5d5 +- **Commit**: evaluator 7b2fd19, draft PR264 (base f9ce6e8); production runtime 32ce5d5 ## Question @@ -40,3 +40,20 @@ Raw logs: /tmp/quality263-red.log, /tmp/quality263-full.log; fixtures/results un ## Conclusion This small synthetic set can expose paired regressions but cannot establish general model quality or statistical equivalence. The compatibility preference question remains unanswered; exact token failures are not waived. + +## First completed paired run + +Qwen3-0.6B, 36 paired cases: nativeGPU32correct/36validformat; RNGD→GPU34correct/36validformat. Correct→wrong0, wrong→correct2; exacttoken34/36. This is a structured synthetic task result, not a general model quality advantage. Raw: quality263-qwen06b-{native-gpu,rngd-gpu}.jsonl and quality263-qwen06b-gpu-summary.json under analysis/serving-matrix-201. Other models remain in the active sequential queue. + +## ATOM execution + +Owned quality263 Pod allocated one ATOM and one RNGD through device plugins; no existing Pod modified. All36 stored prompts are re-tokenized with the remote model tokenizer and asserted equal to frozen input IDs before inference. B1, context9216, cacheoff, normalEOS, max128output; nativeATOM and liveRNGD→ATOM run sequentially with existing compiled artifacts. Runtime3e16c33; kv_blocks3 except KONI1; Gemma global225 artifacts. Queue deletes only its own Pod in finally. + +Local exact orchestrator commands: +```bash +/home/gspark/venvs/pt213/bin/python analysis/serving-matrix-201/run_quality263_gpu.py > analysis/serving-matrix-201/quality263-gpu-queue.log 2>&1 +/home/gspark/venvs/pt213/bin/python analysis/serving-matrix-201/run_quality263_atom_queue.py > analysis/serving-matrix-201/quality263-atom-queue.log 2>&1 +``` +Per-route argv/environment and raw HTTP outputs are retained; first-stage ATOM tokenizer equality passed. No final ATOM scores yet. + +Qwen3-4B GPU pair completed:36/36 answers and36/36 JSON format on both routes; exacttoken36/36; correct→wrong0 and wrong→correct0. Prior long-free-generation mismatch in#256 remains open. From d8e6d0a04e344226fdb7cf20daa56826392000e3 Mon Sep 17 00:00:00 2001 From: gspark Date: Sat, 24 Oct 2026 05:39:16 +0900 Subject: [PATCH 3/7] test: publish all four GPU quality pairs and checkpoint identity audit --- opt_prime/examples/pd/http_matrix/README.md | 13 + opt_prime/examples/pd/http_matrix/STATUS.md | 2 + .../pd/http_matrix/evidence263/hash_models.py | 21 + .../quality263-gemma-native-gpu-run.json | 43 + .../quality263-gemma-native-gpu-server.log | 3 + .../quality263-gemma-native-gpu.jsonl | 36 + .../quality263-gemma-rngd-gpu-run.json | 47 + .../quality263-gemma-rngd-gpu-server.log | 5 + .../quality263-gemma-rngd-gpu.jsonl | 36 + .../quality263-koni-native-gpu-run.json | 43 + .../quality263-koni-native-gpu-server.log | 4 + .../quality263-koni-native-gpu.jsonl | 36 + .../quality263-koni-rngd-gpu-run.json | 47 + .../quality263-koni-rngd-gpu-server.log | 12 + .../quality263-koni-rngd-gpu.jsonl | 36 + .../quality263-local-model-hashes.json | 106 +++ .../quality263-model-identity.json | 30 + .../quality263-qwen06b-native-atom.jsonl | 36 + .../quality263-qwen06b-native-atom.log | 37 + .../quality263-qwen06b-native-atom.run.json | 16 + .../quality263-qwen06b-rngd-atom.jsonl | 36 + .../quality263-qwen06b-rngd-atom.log | 37 + ...uality263-qwen06b-rngd-atom.producer.jsonl | 36 + .../quality263-qwen06b-rngd-atom.run.json | 16 + .../quality263-qwen4b-native-atom.jsonl | 36 + .../quality263-qwen4b-native-atom.log | 37 + .../quality263-qwen4b-native-atom.run.json | 16 + .../quality263-remote-model-hashes.json | 106 +++ .../evidence263/quality263-summary.json | 894 ++++++++++++++++++ .../evidence263/summarize_quality263.py | 64 ++ .../2026-10-24-heldout-quality263.md | 19 + 31 files changed, 1906 insertions(+) create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/hash_models.py create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-native-gpu-run.json create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-native-gpu-server.log create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-native-gpu.jsonl create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-rngd-gpu-run.json create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-rngd-gpu-server.log create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-rngd-gpu.jsonl create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-native-gpu-run.json create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-native-gpu-server.log create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-native-gpu.jsonl create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-rngd-gpu-run.json create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-rngd-gpu-server.log create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-rngd-gpu.jsonl create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-local-model-hashes.json create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-model-identity.json create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen06b-native-atom.jsonl create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen06b-native-atom.log create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen06b-native-atom.run.json create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen06b-rngd-atom.jsonl create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen06b-rngd-atom.log create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen06b-rngd-atom.producer.jsonl create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen06b-rngd-atom.run.json create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen4b-native-atom.jsonl create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen4b-native-atom.log create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen4b-native-atom.run.json create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-remote-model-hashes.json create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-summary.json create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/summarize_quality263.py diff --git a/opt_prime/examples/pd/http_matrix/README.md b/opt_prime/examples/pd/http_matrix/README.md index 2c243a39..a7834601 100644 --- a/opt_prime/examples/pd/http_matrix/README.md +++ b/opt_prime/examples/pd/http_matrix/README.md @@ -77,3 +77,16 @@ structured-answer adherence, not unrestricted semantic quality. `compare` requires matching case sets and input hashes and reports both correct→wrong and wrong→correct transitions plus exact token matches. Do not erase existing strict token failures or infer general equivalence from these 36 synthetic cases. + +Reproduce the recorded partial audit after unpacking the frozen inputs: + +```bash +gzip -dk evidence263/*-fixtures.json.gz +python evidence263/summarize_quality263.py +``` + +The auditor recomputes scores, checks fixture hashes and case correspondence, +and leaves uncollected routes explicit. Gemma's strict score is zero on both +GPU routes because outputs contain Markdown code fences; this does not mean +zero content accuracy. Its four changed pairs comprise three whitespace changes +and one `null`→correct-city answer. No scoring rules were changed after inference. diff --git a/opt_prime/examples/pd/http_matrix/STATUS.md b/opt_prime/examples/pd/http_matrix/STATUS.md index 08ccbc00..f8365472 100644 --- a/opt_prime/examples/pd/http_matrix/STATUS.md +++ b/opt_prime/examples/pd/http_matrix/STATUS.md @@ -53,3 +53,5 @@ Allfour live GPU isolation runs pass: active512-token decode survives other-requ #263 evaluator verification:10focused passed; full prior collection533passed/15skipped/253subtests. Frozen fixtures hashed before inference; GPU route queue started, hardware results pending. #263 first GPU pairs complete: Qwen0.6 native32/36 versus RNGD34/36, no paired regression,34/36 token equality; Qwen4 both36/36, alltokens equal. KONI/Gemma and ATOM queues still running; old long-output failures are retained. + +#263 all288 GPU responses complete; structured scores native/RNGD:Q06 32/34,Q4 36/36,KONI31/31,Gemma0/0 (code fences, not zero content accuracy). Exacttoken pairs34/36,36/36,35/36,32/36. Q06 ATOM pair33/33 scores and36/36 tokens equal; Q4nativeATOM36/36. All source weight shards/config/tokenizer SHA256 match between hosts. Remaining fiveATOM routes active. diff --git a/opt_prime/examples/pd/http_matrix/evidence263/hash_models.py b/opt_prime/examples/pd/http_matrix/evidence263/hash_models.py new file mode 100644 index 00000000..b521b493 --- /dev/null +++ b/opt_prime/examples/pd/http_matrix/evidence263/hash_models.py @@ -0,0 +1,21 @@ +import hashlib,json,sys +from pathlib import Path +hub=Path.home()/'.cache/huggingface/hub' +models=[('qwen06b','Qwen--Qwen3-0.6B',None),('qwen4b','Qwen--Qwen3-4B','Qwen3-4B'),('koni','KISTI-KONI--KONI-Llama3.1-8B-Instruct-20241024','KONI-8B'),('gemma','unsloth--gemma-3-1b-it','Gemma3-1B')] +result={} +for label,repo,alias in models: + cache=hub/('models--'+repo) + revision=(cache/'refs/main').read_text().strip() if (cache/'refs/main').exists() else None + if sys.argv[1]=='local' and alias: folder=Path('/tmp/serving-matrix-models')/alias + else: folder=cache/'snapshots'/revision + files=sorted(set(folder.glob('*.safetensors')) | {folder/'config.json',folder/'generation_config.json',folder/'tokenizer.json'}) + row={} + for p in files: + if not p.exists():continue + h=hashlib.sha256() + with p.open('rb') as stream: + for chunk in iter(lambda:stream.read(8*1024*1024),b''): h.update(chunk) + row[p.name]={'bytes':p.stat().st_size,'sha256':h.hexdigest()} + assert any(name.endswith('.safetensors') for name in row) + result[label]={'revision_ref':revision,'files':row} +print(json.dumps(result,indent=2)) diff --git a/opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-native-gpu-run.json b/opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-native-gpu-run.json new file mode 100644 index 00000000..1991387e --- /dev/null +++ b/opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-native-gpu-run.json @@ -0,0 +1,43 @@ +{ + "label": "gemma", + "route": "native-gpu", + "argv": [ + "/home/gspark/venvs/pt213/bin/python", + "-c", + "import sys,importlib.util,runpy\ns=importlib.util.spec_from_file_location('opt_prime.accel.serve.llmd','/tmp/llmd240.py')\nm=importlib.util.module_from_spec(s);sys.modules[s.name]=m;s.loader.exec_module(m)\nrunpy.run_path('/home/gspark/workspace/aicomp-dense-hybrid-258/opt_prime/examples/pd/serve_batched.py',run_name='__main__')\n", + "--model", + "/tmp/serving-matrix-models/Gemma3-1B", + "--device", + "cuda:0", + "--dtype", + "bfloat16", + "--slots", + "4", + "--max-context", + "9216", + "--max-output", + "128", + "--kv", + "paged", + "--page-size", + "64", + "--chunk", + "256", + "--port", + "40861" + ], + "env": { + "OMP_NUM_THREADS": "4", + "MKL_NUM_THREADS": "4", + "TORCH_DEVICE_BACKEND_AUTOLOAD": "0", + "HF_HUB_OFFLINE": "1", + "TRANSFORMERS_OFFLINE": "1", + "PYTHONPATH": "/tmp/rngd-gpu201-src:/home/gspark/workspace/aicomp-live-pd-234/opt_prime/examples/pd/serving_matrix:/home/gspark/workspace/aicomp-dense-hybrid-258/opt_prime" + }, + "runtime_commit": "32ce5d5", + "stage": "complete", + "cases": 36, + "answer_correct": 0, + "format_correct": 0, + "server_exit": -15 +} diff --git a/opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-native-gpu-server.log b/opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-native-gpu-server.log new file mode 100644 index 00000000..e9629633 --- /dev/null +++ b/opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-native-gpu-server.log @@ -0,0 +1,3 @@ + +Loading weights: 0%| | 0/340 [00:00 Date: Sat, 24 Oct 2026 05:54:16 +0900 Subject: [PATCH 4/7] test: verify Qwen3-4B across all four hardware routes --- opt_prime/examples/pd/http_matrix/STATUS.md | 2 + .../quality263-qwen4b-rngd-atom.jsonl | 36 +++++++++++ .../quality263-qwen4b-rngd-atom.log | 37 +++++++++++ ...quality263-qwen4b-rngd-atom.producer.jsonl | 36 +++++++++++ .../quality263-qwen4b-rngd-atom.run.json | 16 +++++ .../evidence263/quality263-summary.json | 61 ++++++++++++++++++- .../evidence263/research-events.jsonl | 6 ++ .../2026-10-24-heldout-quality263.md | 2 + 8 files changed, 195 insertions(+), 1 deletion(-) create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen4b-rngd-atom.jsonl create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen4b-rngd-atom.log create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen4b-rngd-atom.producer.jsonl create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen4b-rngd-atom.run.json diff --git a/opt_prime/examples/pd/http_matrix/STATUS.md b/opt_prime/examples/pd/http_matrix/STATUS.md index f8365472..89181459 100644 --- a/opt_prime/examples/pd/http_matrix/STATUS.md +++ b/opt_prime/examples/pd/http_matrix/STATUS.md @@ -55,3 +55,5 @@ Allfour live GPU isolation runs pass: active512-token decode survives other-requ #263 first GPU pairs complete: Qwen0.6 native32/36 versus RNGD34/36, no paired regression,34/36 token equality; Qwen4 both36/36, alltokens equal. KONI/Gemma and ATOM queues still running; old long-output failures are retained. #263 all288 GPU responses complete; structured scores native/RNGD:Q06 32/34,Q4 36/36,KONI31/31,Gemma0/0 (code fences, not zero content accuracy). Exacttoken pairs34/36,36/36,35/36,32/36. Q06 ATOM pair33/33 scores and36/36 tokens equal; Q4nativeATOM36/36. All source weight shards/config/tokenizer SHA256 match between hosts. Remaining fiveATOM routes active. + +#263 Qwen4allfour routes complete:each36/36 structured answers and all36 token sequences equal acrossGPU/RNGD/ATOM. KONI/GemmaATOM pending; historical#256 remains open. diff --git a/opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen4b-rngd-atom.jsonl b/opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen4b-rngd-atom.jsonl new file mode 100644 index 00000000..7c724c00 --- /dev/null +++ b/opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen4b-rngd-atom.jsonl @@ -0,0 +1,36 @@ +{"case": "arithmetic-0-0", "route": "rngd-atom", "input_sha256": "ffb0cb6f0a0322a9f0b0272419627e2beeeb9b52fbb50bcc2bdda6b693accb6b", "input_tokens": 40, "expected": "96", "text": "{\n \"answer\": \"96\"\n}", "tokens": [515, 220, 330, 9217, 788, 330, 24, 21, 698, 92, 151645], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-0-0", "object": "text_completion", "created": 1792788603, "model": "/tmp/serving-matrix-models/Qwen3-4B", "choices": [{"index": 0, "text": "{\n \"answer\": \"96\"\n}", "token_ids": [515, 220, 330, 9217, 788, 330, 24, 21, 698, 92, 151645], "finish_reason": "stop"}], "usage": {"prompt_tokens": 40, "completion_tokens": 11, "total_tokens": 51}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 3.882162579917349} +{"case": "passkey-0-0", "route": "rngd-atom", "input_sha256": "d83f23a8895d759da4005de77773266f03fb8edafe5426d1aa93e06e7f89d464", "input_tokens": 58, "expected": "34HYL6W8", "text": "{\"answer\": \"34HYL6W8\"}", "tokens": [4913, 9217, 788, 330, 18, 19, 48852, 43, 21, 54, 23, 9207, 151645], "finish_reason": "stop", "response": {"id": "quality263-passkey-0-0", "object": "text_completion", "created": 1792788607, "model": "/tmp/serving-matrix-models/Qwen3-4B", "choices": [{"index": 0, "text": "{\"answer\": \"34HYL6W8\"}", "token_ids": [4913, 9217, 788, 330, 18, 19, 48852, 43, 21, 54, 23, 9207, 151645], "finish_reason": "stop"}], "usage": {"prompt_tokens": 58, "completion_tokens": 13, "total_tokens": 71}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 3.546389932045713} +{"case": "korean_table-0-0", "route": "rngd-atom", "input_sha256": "87b678f37aa562ce8825a85ddcd468c0021f05855c288f850b23469a4b4866db", "input_tokens": 77, "expected": "춘천", "text": "{\"answer\": \"춘천\"}", "tokens": [4913, 9217, 788, 330, 144079, 129034, 9207, 151645], "finish_reason": "stop", "response": {"id": "quality263-korean_table-0-0", "object": "text_completion", "created": 1792788610, "model": "/tmp/serving-matrix-models/Qwen3-4B", "choices": [{"index": 0, "text": "{\"answer\": \"춘천\"}", "token_ids": [4913, 9217, 788, 330, 144079, 129034, 9207, 151645], "finish_reason": "stop"}], "usage": {"prompt_tokens": 77, "completion_tokens": 8, "total_tokens": 85}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 3.372935126069933} +{"case": "arithmetic-1-0", "route": "rngd-atom", "input_sha256": "e8aa12a2c146e56722bcd58c8f05cfc06c2f943bbeb21697852e7691ba63386d", "input_tokens": 40, "expected": "83", "text": "{\n \"answer\": \"83\"\n}", "tokens": [515, 220, 330, 9217, 788, 330, 23, 18, 698, 92, 151645], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-1-0", "object": "text_completion", "created": 1792788614, "model": "/tmp/serving-matrix-models/Qwen3-4B", "choices": [{"index": 0, "text": "{\n \"answer\": \"83\"\n}", "token_ids": [515, 220, 330, 9217, 788, 330, 23, 18, 698, 92, 151645], "finish_reason": "stop"}], "usage": {"prompt_tokens": 40, "completion_tokens": 11, "total_tokens": 51}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 3.48351697192993} +{"case": "passkey-1-0", "route": "rngd-atom", "input_sha256": "8aafbecb722292249bd8b14116666b7013a1b2e8a3c7dc07ca5a44a935f36b54", "input_tokens": 55, "expected": "CNBKEFR7", "text": "{\"answer\": \"CNBKEFR7\"}", "tokens": [4913, 9217, 788, 330, 28668, 33, 3390, 10504, 22, 9207, 151645], "finish_reason": "stop", "response": {"id": "quality263-passkey-1-0", "object": "text_completion", "created": 1792788617, "model": "/tmp/serving-matrix-models/Qwen3-4B", "choices": [{"index": 0, "text": "{\"answer\": \"CNBKEFR7\"}", "token_ids": [4913, 9217, 788, 330, 28668, 33, 3390, 10504, 22, 9207, 151645], "finish_reason": "stop"}], "usage": {"prompt_tokens": 55, "completion_tokens": 11, "total_tokens": 66}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 3.4911097639705986} +{"case": "korean_table-1-0", "route": "rngd-atom", "input_sha256": "d3463b07556c6f05ee2cc92c85345f4111cbcabb3287e6a3271f6bf111daa81b", "input_tokens": 77, "expected": "목포", "text": "{\"answer\": \"목포\"}", "tokens": [4913, 9217, 788, 330, 87608, 128808, 9207, 151645], "finish_reason": "stop", "response": {"id": "quality263-korean_table-1-0", "object": "text_completion", "created": 1792788621, "model": "/tmp/serving-matrix-models/Qwen3-4B", "choices": [{"index": 0, "text": "{\"answer\": \"목포\"}", "token_ids": [4913, 9217, 788, 330, 87608, 128808, 9207, 151645], "finish_reason": "stop"}], "usage": {"prompt_tokens": 77, "completion_tokens": 8, "total_tokens": 85}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 3.367778643965721} +{"case": "arithmetic-2-0", "route": "rngd-atom", "input_sha256": "03a53c7187665c581099ba9a5fa766beb02786bbeaa3feb904dd5be11cbe1dc2", "input_tokens": 40, "expected": "43", "text": "{\n \"answer\": \"43\"\n}", "tokens": [515, 220, 330, 9217, 788, 330, 19, 18, 698, 92, 151645], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-2-0", "object": "text_completion", "created": 1792788624, "model": "/tmp/serving-matrix-models/Qwen3-4B", "choices": [{"index": 0, "text": "{\n \"answer\": \"43\"\n}", "token_ids": [515, 220, 330, 9217, 788, 330, 19, 18, 698, 92, 151645], "finish_reason": "stop"}], "usage": {"prompt_tokens": 40, "completion_tokens": 11, "total_tokens": 51}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 3.398250567028299} +{"case": "passkey-2-0", "route": "rngd-atom", "input_sha256": "a7d42adb12a61610212ad3ccab6ecebba2f1e7d0d78975db50b535e919f2e85c", "input_tokens": 55, "expected": "WMLLRJUL", "text": "{\"answer\": \"WMLLRJUL\"}", "tokens": [4913, 9217, 788, 330, 54, 2668, 20117, 41, 1094, 9207, 151645], "finish_reason": "stop", "response": {"id": "quality263-passkey-2-0", "object": "text_completion", "created": 1792788628, "model": "/tmp/serving-matrix-models/Qwen3-4B", "choices": [{"index": 0, "text": "{\"answer\": \"WMLLRJUL\"}", "token_ids": [4913, 9217, 788, 330, 54, 2668, 20117, 41, 1094, 9207, 151645], "finish_reason": "stop"}], "usage": {"prompt_tokens": 55, "completion_tokens": 11, "total_tokens": 66}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 3.413239203975536} +{"case": "korean_table-2-0", "route": "rngd-atom", "input_sha256": "9fa1176c67e8742628802495a9590ee4fedf0247e8394e9626296d09ca9b5700", "input_tokens": 77, "expected": "전주", "text": "{\"answer\": \"전주\"}", "tokens": [4913, 9217, 788, 330, 65865, 54330, 9207, 151645], "finish_reason": "stop", "response": {"id": "quality263-korean_table-2-0", "object": "text_completion", "created": 1792788631, "model": "/tmp/serving-matrix-models/Qwen3-4B", "choices": [{"index": 0, "text": "{\"answer\": \"전주\"}", "token_ids": [4913, 9217, 788, 330, 65865, 54330, 9207, 151645], "finish_reason": "stop"}], "usage": {"prompt_tokens": 77, "completion_tokens": 8, "total_tokens": 85}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 3.3894951819675043} +{"case": "arithmetic-3-0", "route": "rngd-atom", "input_sha256": "ed6de815520fd4821b88ed18f5163b5268db1d1daa8dd205d27937f993d3df2d", "input_tokens": 40, "expected": "63", "text": "{\n \"answer\": \"63\"\n}", "tokens": [515, 220, 330, 9217, 788, 330, 21, 18, 698, 92, 151645], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-3-0", "object": "text_completion", "created": 1792788634, "model": "/tmp/serving-matrix-models/Qwen3-4B", "choices": [{"index": 0, "text": "{\n \"answer\": \"63\"\n}", "token_ids": [515, 220, 330, 9217, 788, 330, 21, 18, 698, 92, 151645], "finish_reason": "stop"}], "usage": {"prompt_tokens": 40, "completion_tokens": 11, "total_tokens": 51}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 3.4835062390193343} +{"case": "passkey-3-0", "route": "rngd-atom", "input_sha256": "258cd6bbd0ed03d3b1efb4df15ad4950c2e51ea0cec80685f475026c0da08690", "input_tokens": 56, "expected": "SBYZEGE7", "text": "{\"answer\": \"SBYZEGE7\"}", "tokens": [4913, 9217, 788, 330, 16680, 40209, 9376, 36, 22, 9207, 151645], "finish_reason": "stop", "response": {"id": "quality263-passkey-3-0", "object": "text_completion", "created": 1792788638, "model": "/tmp/serving-matrix-models/Qwen3-4B", "choices": [{"index": 0, "text": "{\"answer\": \"SBYZEGE7\"}", "token_ids": [4913, 9217, 788, 330, 16680, 40209, 9376, 36, 22, 9207, 151645], "finish_reason": "stop"}], "usage": {"prompt_tokens": 56, "completion_tokens": 11, "total_tokens": 67}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 3.381225080927834} +{"case": "korean_table-3-0", "route": "rngd-atom", "input_sha256": "3d32176f69c72579ce4b8bfe5e7197e20139d98ffcd6d9a28f3354d72061cf48", "input_tokens": 77, "expected": "속초", "text": "{\"answer\": \"속초\"}", "tokens": [4913, 9217, 788, 330, 126299, 132618, 9207, 151645], "finish_reason": "stop", "response": {"id": "quality263-korean_table-3-0", "object": "text_completion", "created": 1792788641, "model": "/tmp/serving-matrix-models/Qwen3-4B", "choices": [{"index": 0, "text": "{\"answer\": \"속초\"}", "token_ids": [4913, 9217, 788, 330, 126299, 132618, 9207, 151645], "finish_reason": "stop"}], "usage": {"prompt_tokens": 77, "completion_tokens": 8, "total_tokens": 85}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 3.232552225003019} +{"case": "arithmetic-0-2048", "route": "rngd-atom", "input_sha256": "7bcbba5f5870664833591b27c524ab712f1e5ec6d37d5521311995815f0779e5", "input_tokens": 2042, "expected": "96", "text": "{\"answer\": \"96\"}", "tokens": [4913, 9217, 788, 330, 24, 21, 9207, 151645], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-0-2048", "object": "text_completion", "created": 1792788645, "model": "/tmp/serving-matrix-models/Qwen3-4B", "choices": [{"index": 0, "text": "{\"answer\": \"96\"}", "token_ids": [4913, 9217, 788, 330, 24, 21, 9207, 151645], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2042, "completion_tokens": 8, "total_tokens": 2050}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 3.854236076003872} +{"case": "passkey-0-2048", "route": "rngd-atom", "input_sha256": "f4d815e8749c966b6dd1751e7a03d4cbe8ec25c8a793c4eb8e33c1e93c8a9d8a", "input_tokens": 2046, "expected": "34HYL6W8", "text": "{\"answer\": \"34HYL6W8\"}", "tokens": [4913, 9217, 788, 330, 18, 19, 48852, 43, 21, 54, 23, 9207, 151645], "finish_reason": "stop", "response": {"id": "quality263-passkey-0-2048", "object": "text_completion", "created": 1792788649, "model": "/tmp/serving-matrix-models/Qwen3-4B", "choices": [{"index": 0, "text": "{\"answer\": \"34HYL6W8\"}", "token_ids": [4913, 9217, 788, 330, 18, 19, 48852, 43, 21, 54, 23, 9207, 151645], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2046, "completion_tokens": 13, "total_tokens": 2059}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 3.972199884010479} +{"case": "korean_table-0-2048", "route": "rngd-atom", "input_sha256": "4a9d9a40bd5a22ddf7a9723bce0bebcac37894732cd94dc8dedada4d96a753a8", "input_tokens": 2037, "expected": "춘천", "text": "{\"answer\": \"춘천\"}", "tokens": [4913, 9217, 788, 330, 144079, 129034, 9207, 151645], "finish_reason": "stop", "response": {"id": "quality263-korean_table-0-2048", "object": "text_completion", "created": 1792788653, "model": "/tmp/serving-matrix-models/Qwen3-4B", "choices": [{"index": 0, "text": "{\"answer\": \"춘천\"}", "token_ids": [4913, 9217, 788, 330, 144079, 129034, 9207, 151645], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2037, "completion_tokens": 8, "total_tokens": 2045}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 3.7614943699445575} +{"case": "arithmetic-1-2048", "route": "rngd-atom", "input_sha256": "7af35b97c003792c0665024eb220e355e990835cce815ee3d477f5cd90611b0a", "input_tokens": 2042, "expected": "83", "text": "{\"answer\": \"83\"}", "tokens": [4913, 9217, 788, 330, 23, 18, 9207, 151645], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-1-2048", "object": "text_completion", "created": 1792788656, "model": "/tmp/serving-matrix-models/Qwen3-4B", "choices": [{"index": 0, "text": "{\"answer\": \"83\"}", "token_ids": [4913, 9217, 788, 330, 23, 18, 9207, 151645], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2042, "completion_tokens": 8, "total_tokens": 2050}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 3.6085749379126355} +{"case": "passkey-1-2048", "route": "rngd-atom", "input_sha256": "d966a9942558a02ce5ec3368c0d29db047d17d66564d1bae8a5062b5d655ee37", "input_tokens": 2043, "expected": "CNBKEFR7", "text": "{\"answer\": \"CNBKEFR7\"}", "tokens": [4913, 9217, 788, 330, 28668, 33, 3390, 10504, 22, 9207, 151645], "finish_reason": "stop", "response": {"id": "quality263-passkey-1-2048", "object": "text_completion", "created": 1792788660, "model": "/tmp/serving-matrix-models/Qwen3-4B", "choices": [{"index": 0, "text": "{\"answer\": \"CNBKEFR7\"}", "token_ids": [4913, 9217, 788, 330, 28668, 33, 3390, 10504, 22, 9207, 151645], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2043, "completion_tokens": 11, "total_tokens": 2054}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 3.699291867087595} +{"case": "korean_table-1-2048", "route": "rngd-atom", "input_sha256": "702e8123afa0d7129d8ffa8223264250e71a018f4a022ea6c1d2784d79b3f4e7", "input_tokens": 2037, "expected": "목포", "text": "{\"answer\": \"목포\"}", "tokens": [4913, 9217, 788, 330, 87608, 128808, 9207, 151645], "finish_reason": "stop", "response": {"id": "quality263-korean_table-1-2048", "object": "text_completion", "created": 1792788664, "model": "/tmp/serving-matrix-models/Qwen3-4B", "choices": [{"index": 0, "text": "{\"answer\": \"목포\"}", "token_ids": [4913, 9217, 788, 330, 87608, 128808, 9207, 151645], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2037, "completion_tokens": 8, "total_tokens": 2045}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 3.679842896061018} +{"case": "arithmetic-2-2048", "route": "rngd-atom", "input_sha256": "e48e2dfcd7102ba46de2b10997572f1627d8accd5a4ea624146925def9cad372", "input_tokens": 2042, "expected": "43", "text": "{\"answer\": \"43\"}", "tokens": [4913, 9217, 788, 330, 19, 18, 9207, 151645], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-2-2048", "object": "text_completion", "created": 1792788667, "model": "/tmp/serving-matrix-models/Qwen3-4B", "choices": [{"index": 0, "text": "{\"answer\": \"43\"}", "token_ids": [4913, 9217, 788, 330, 19, 18, 9207, 151645], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2042, "completion_tokens": 8, "total_tokens": 2050}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 3.715455767000094} +{"case": "passkey-2-2048", "route": "rngd-atom", "input_sha256": "6dd6187d1ec7862f671ea402809ab82c15da4a06899efe45190508d892a17a52", "input_tokens": 2043, "expected": "WMLLRJUL", "text": "{\"answer\": \"WMLLRJUL\"}", "tokens": [4913, 9217, 788, 330, 54, 2668, 20117, 41, 1094, 9207, 151645], "finish_reason": "stop", "response": {"id": "quality263-passkey-2-2048", "object": "text_completion", "created": 1792788671, "model": "/tmp/serving-matrix-models/Qwen3-4B", "choices": [{"index": 0, "text": "{\"answer\": \"WMLLRJUL\"}", "token_ids": [4913, 9217, 788, 330, 54, 2668, 20117, 41, 1094, 9207, 151645], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2043, "completion_tokens": 11, "total_tokens": 2054}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 3.8111011700239033} +{"case": "korean_table-2-2048", "route": "rngd-atom", "input_sha256": "855c01047a9a990b9419d2311dbb2868dfdbf298305c47375147250d3c8a2bf2", "input_tokens": 2037, "expected": "전주", "text": "{\"answer\": \"전주\"}", "tokens": [4913, 9217, 788, 330, 65865, 54330, 9207, 151645], "finish_reason": "stop", "response": {"id": "quality263-korean_table-2-2048", "object": "text_completion", "created": 1792788675, "model": "/tmp/serving-matrix-models/Qwen3-4B", "choices": [{"index": 0, "text": "{\"answer\": \"전주\"}", "token_ids": [4913, 9217, 788, 330, 65865, 54330, 9207, 151645], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2037, "completion_tokens": 8, "total_tokens": 2045}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 3.741075783967972} +{"case": "arithmetic-3-2048", "route": "rngd-atom", "input_sha256": "36ba7ee2977c114d6f60d8de1bbc72ad29dba1e3fddf872d29b736e175d330e7", "input_tokens": 2042, "expected": "63", "text": "{\"answer\": \"63\"}", "tokens": [4913, 9217, 788, 330, 21, 18, 9207, 151645], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-3-2048", "object": "text_completion", "created": 1792788679, "model": "/tmp/serving-matrix-models/Qwen3-4B", "choices": [{"index": 0, "text": "{\"answer\": \"63\"}", "token_ids": [4913, 9217, 788, 330, 21, 18, 9207, 151645], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2042, "completion_tokens": 8, "total_tokens": 2050}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 3.787856315029785} +{"case": "passkey-3-2048", "route": "rngd-atom", "input_sha256": "bcfde1337228285d1933ed16092643419ff6d2538831ae243b954cd58ca3bfac", "input_tokens": 2044, "expected": "SBYZEGE7", "text": "{\"answer\": \"SBYZEGE7\"}", "tokens": [4913, 9217, 788, 330, 16680, 40209, 9376, 36, 22, 9207, 151645], "finish_reason": "stop", "response": {"id": "quality263-passkey-3-2048", "object": "text_completion", "created": 1792788683, "model": "/tmp/serving-matrix-models/Qwen3-4B", "choices": [{"index": 0, "text": "{\"answer\": \"SBYZEGE7\"}", "token_ids": [4913, 9217, 788, 330, 16680, 40209, 9376, 36, 22, 9207, 151645], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2044, "completion_tokens": 11, "total_tokens": 2055}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 3.8494737721048295} +{"case": "korean_table-3-2048", "route": "rngd-atom", "input_sha256": "0ea54813c513394d5f73929812722bab7bfc68190de8bdcd73c5891854f1020a", "input_tokens": 2037, "expected": "속초", "text": "{\"answer\": \"속초\"}", "tokens": [4913, 9217, 788, 330, 126299, 132618, 9207, 151645], "finish_reason": "stop", "response": {"id": "quality263-korean_table-3-2048", "object": "text_completion", "created": 1792788686, "model": "/tmp/serving-matrix-models/Qwen3-4B", "choices": [{"index": 0, "text": "{\"answer\": \"속초\"}", "token_ids": [4913, 9217, 788, 330, 126299, 132618, 9207, 151645], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2037, "completion_tokens": 8, "total_tokens": 2045}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 3.7009095799876377} +{"case": "arithmetic-0-8192", "route": "rngd-atom", "input_sha256": "c26831baaa88d48f9a0d30d69ef0d8d028687558b703e9880473ae7480cb6713", "input_tokens": 8188, "expected": "96", "text": "{\"answer\": \"96\"}", "tokens": [4913, 9217, 788, 330, 24, 21, 9207, 151645], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-0-8192", "object": "text_completion", "created": 1792788692, "model": "/tmp/serving-matrix-models/Qwen3-4B", "choices": [{"index": 0, "text": "{\"answer\": \"96\"}", "token_ids": [4913, 9217, 788, 330, 24, 21, 9207, 151645], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8188, "completion_tokens": 8, "total_tokens": 8196}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 5.518850888940506} +{"case": "passkey-0-8192", "route": "rngd-atom", "input_sha256": "4f9c677a78568341f6fdb5fc7b52bfd698d808aa5c87e7a39c89aca1057374d6", "input_tokens": 8192, "expected": "34HYL6W8", "text": "{\"answer\": \"34HYL6W8\"}", "tokens": [4913, 9217, 788, 330, 18, 19, 48852, 43, 21, 54, 23, 9207, 151645], "finish_reason": "stop", "response": {"id": "quality263-passkey-0-8192", "object": "text_completion", "created": 1792788697, "model": "/tmp/serving-matrix-models/Qwen3-4B", "choices": [{"index": 0, "text": "{\"answer\": \"34HYL6W8\"}", "token_ids": [4913, 9217, 788, 330, 18, 19, 48852, 43, 21, 54, 23, 9207, 151645], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8192, "completion_tokens": 13, "total_tokens": 8205}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 5.066118787974119} +{"case": "korean_table-0-8192", "route": "rngd-atom", "input_sha256": "35d9b8319110711c12d392b0a3797c360aa274cb6d5c8a528b335e5686177578", "input_tokens": 8183, "expected": "춘천", "text": "{\"answer\": \"춘천\"}", "tokens": [4913, 9217, 788, 330, 144079, 129034, 9207, 151645], "finish_reason": "stop", "response": {"id": "quality263-korean_table-0-8192", "object": "text_completion", "created": 1792788702, "model": "/tmp/serving-matrix-models/Qwen3-4B", "choices": [{"index": 0, "text": "{\"answer\": \"춘천\"}", "token_ids": [4913, 9217, 788, 330, 144079, 129034, 9207, 151645], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8183, "completion_tokens": 8, "total_tokens": 8191}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 5.0080762499710545} +{"case": "arithmetic-1-8192", "route": "rngd-atom", "input_sha256": "c987e11e00b7680c5280d64a16e574aa8b10756e855f1c1304fdc1598e1dc15a", "input_tokens": 8188, "expected": "83", "text": "{\"answer\": \"83\"}", "tokens": [4913, 9217, 788, 330, 23, 18, 9207, 151645], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-1-8192", "object": "text_completion", "created": 1792788707, "model": "/tmp/serving-matrix-models/Qwen3-4B", "choices": [{"index": 0, "text": "{\"answer\": \"83\"}", "token_ids": [4913, 9217, 788, 330, 23, 18, 9207, 151645], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8188, "completion_tokens": 8, "total_tokens": 8196}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 4.980620859074406} +{"case": "passkey-1-8192", "route": "rngd-atom", "input_sha256": "f6851b3f62ad7f00a6885cb62364ecae1f5b4a635fe7d428cc3b87ea1ab53542", "input_tokens": 8189, "expected": "CNBKEFR7", "text": "{\"answer\": \"CNBKEFR7\"}", "tokens": [4913, 9217, 788, 330, 28668, 33, 3390, 10504, 22, 9207, 151645], "finish_reason": "stop", "response": {"id": "quality263-passkey-1-8192", "object": "text_completion", "created": 1792788712, "model": "/tmp/serving-matrix-models/Qwen3-4B", "choices": [{"index": 0, "text": "{\"answer\": \"CNBKEFR7\"}", "token_ids": [4913, 9217, 788, 330, 28668, 33, 3390, 10504, 22, 9207, 151645], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8189, "completion_tokens": 11, "total_tokens": 8200}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 5.130263008060865} +{"case": "korean_table-1-8192", "route": "rngd-atom", "input_sha256": "c807115742aae9d7d25dbead224dca179c1e25fd3ece974e67e96bf0fd414230", "input_tokens": 8183, "expected": "목포", "text": "{\"answer\": \"목포\"}", "tokens": [4913, 9217, 788, 330, 87608, 128808, 9207, 151645], "finish_reason": "stop", "response": {"id": "quality263-korean_table-1-8192", "object": "text_completion", "created": 1792788717, "model": "/tmp/serving-matrix-models/Qwen3-4B", "choices": [{"index": 0, "text": "{\"answer\": \"목포\"}", "token_ids": [4913, 9217, 788, 330, 87608, 128808, 9207, 151645], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8183, "completion_tokens": 8, "total_tokens": 8191}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 4.9946485430700704} +{"case": "arithmetic-2-8192", "route": "rngd-atom", "input_sha256": "a778b21536e1907c77006d72900ddc8cb83546efbb72304c0bb321b4a22861dd", "input_tokens": 8188, "expected": "43", "text": "{\"answer\": \"43\"}", "tokens": [4913, 9217, 788, 330, 19, 18, 9207, 151645], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-2-8192", "object": "text_completion", "created": 1792788722, "model": "/tmp/serving-matrix-models/Qwen3-4B", "choices": [{"index": 0, "text": "{\"answer\": \"43\"}", "token_ids": [4913, 9217, 788, 330, 19, 18, 9207, 151645], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8188, "completion_tokens": 8, "total_tokens": 8196}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 5.001303678960539} +{"case": "passkey-2-8192", "route": "rngd-atom", "input_sha256": "16d0ab8cc07bbf0459dfc6c7c20f990b85a01915686734f5a94d0c71db1854c0", "input_tokens": 8189, "expected": "WMLLRJUL", "text": "{\"answer\": \"WMLLRJUL\"}", "tokens": [4913, 9217, 788, 330, 54, 2668, 20117, 41, 1094, 9207, 151645], "finish_reason": "stop", "response": {"id": "quality263-passkey-2-8192", "object": "text_completion", "created": 1792788727, "model": "/tmp/serving-matrix-models/Qwen3-4B", "choices": [{"index": 0, "text": "{\"answer\": \"WMLLRJUL\"}", "token_ids": [4913, 9217, 788, 330, 54, 2668, 20117, 41, 1094, 9207, 151645], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8189, "completion_tokens": 11, "total_tokens": 8200}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 5.0877209439640865} +{"case": "korean_table-2-8192", "route": "rngd-atom", "input_sha256": "706cb4ae9b6e2bc3925b61c328e1dd9d4adb1a2ee9572a13c4b21907ee101c53", "input_tokens": 8183, "expected": "전주", "text": "{\"answer\": \"전주\"}", "tokens": [4913, 9217, 788, 330, 65865, 54330, 9207, 151645], "finish_reason": "stop", "response": {"id": "quality263-korean_table-2-8192", "object": "text_completion", "created": 1792788732, "model": "/tmp/serving-matrix-models/Qwen3-4B", "choices": [{"index": 0, "text": "{\"answer\": \"전주\"}", "token_ids": [4913, 9217, 788, 330, 65865, 54330, 9207, 151645], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8183, "completion_tokens": 8, "total_tokens": 8191}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 5.003034144989215} +{"case": "arithmetic-3-8192", "route": "rngd-atom", "input_sha256": "04a07bc40b05d492d2aa9848f05860b22c13bbdf854f4c4fdbe25a1068142bfe", "input_tokens": 8188, "expected": "63", "text": "{\"answer\": \"63\"}", "tokens": [4913, 9217, 788, 330, 21, 18, 9207, 151645], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-3-8192", "object": "text_completion", "created": 1792788737, "model": "/tmp/serving-matrix-models/Qwen3-4B", "choices": [{"index": 0, "text": "{\"answer\": \"63\"}", "token_ids": [4913, 9217, 788, 330, 21, 18, 9207, 151645], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8188, "completion_tokens": 8, "total_tokens": 8196}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 5.0042936590034515} +{"case": "passkey-3-8192", "route": "rngd-atom", "input_sha256": "8676fb55df4df6176ad0cd1ee6f04b336b7a5817253519a76a044b598a22a1cf", "input_tokens": 8190, "expected": "SBYZEGE7", "text": "{\"answer\": \"SBYZEGE7\"}", "tokens": [4913, 9217, 788, 330, 16680, 40209, 9376, 36, 22, 9207, 151645], "finish_reason": "stop", "response": {"id": "quality263-passkey-3-8192", "object": "text_completion", "created": 1792788742, "model": "/tmp/serving-matrix-models/Qwen3-4B", "choices": [{"index": 0, "text": "{\"answer\": \"SBYZEGE7\"}", "token_ids": [4913, 9217, 788, 330, 16680, 40209, 9376, 36, 22, 9207, 151645], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8190, "completion_tokens": 11, "total_tokens": 8201}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 5.0953901330940425} +{"case": "korean_table-3-8192", "route": "rngd-atom", "input_sha256": "30a0867e33fd1fd7133ea78aeeec7fb78964d3e6af3fe60490fa12031ed6c92f", "input_tokens": 8183, "expected": "속초", "text": "{\"answer\": \"속초\"}", "tokens": [4913, 9217, 788, 330, 126299, 132618, 9207, 151645], "finish_reason": "stop", "response": {"id": "quality263-korean_table-3-8192", "object": "text_completion", "created": 1792788747, "model": "/tmp/serving-matrix-models/Qwen3-4B", "choices": [{"index": 0, "text": "{\"answer\": \"속초\"}", "token_ids": [4913, 9217, 788, 330, 126299, 132618, 9207, 151645], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8183, "completion_tokens": 8, "total_tokens": 8191}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 5.109273859998211} diff --git a/opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen4b-rngd-atom.log b/opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen4b-rngd-atom.log new file mode 100644 index 00000000..f6836111 --- /dev/null +++ b/opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen4b-rngd-atom.log @@ -0,0 +1,37 @@ +ATOM_BINDING {"visible_count": 1, "physical": 1, "opened": ["/dev/rbln1"]} +arithmetic-0-0 True True +passkey-0-0 True True +korean_table-0-0 True True +arithmetic-1-0 True True +passkey-1-0 True True +korean_table-1-0 True True +arithmetic-2-0 True True +passkey-2-0 True True +korean_table-2-0 True True +arithmetic-3-0 True True +passkey-3-0 True True +korean_table-3-0 True True +arithmetic-0-2048 True True +passkey-0-2048 True True +korean_table-0-2048 True True +arithmetic-1-2048 True True +passkey-1-2048 True True +korean_table-1-2048 True True +arithmetic-2-2048 True True +passkey-2-2048 True True +korean_table-2-2048 True True +arithmetic-3-2048 True True +passkey-3-2048 True True +korean_table-3-2048 True True +arithmetic-0-8192 True True +passkey-0-8192 True True +korean_table-0-8192 True True +arithmetic-1-8192 True True +passkey-1-8192 True True +korean_table-1-8192 True True +arithmetic-2-8192 True True +passkey-2-8192 True True +korean_table-2-8192 True True +arithmetic-3-8192 True True +passkey-3-8192 True True +korean_table-3-8192 True True diff --git a/opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen4b-rngd-atom.producer.jsonl b/opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen4b-rngd-atom.producer.jsonl new file mode 100644 index 00000000..6cf8842c --- /dev/null +++ b/opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen4b-rngd-atom.producer.jsonl @@ -0,0 +1,36 @@ +{"wire_id": 0, "start_s": 763574.98690571, "end_s": 763578.262459867} +{"wire_id": 1, "start_s": 763578.867244132, "end_s": 763581.931428551} +{"wire_id": 2, "start_s": 763582.413040548, "end_s": 763585.482733256} +{"wire_id": 3, "start_s": 763585.786492108, "end_s": 763588.865058542} +{"wire_id": 4, "start_s": 763589.270416915, "end_s": 763592.352261134} +{"wire_id": 5, "start_s": 763592.761833981, "end_s": 763595.841383887} +{"wire_id": 6, "start_s": 763596.129968559, "end_s": 763599.125206727} +{"wire_id": 7, "start_s": 763599.528395378, "end_s": 763602.531771312} +{"wire_id": 8, "start_s": 763602.941960227, "end_s": 763606.028275264} +{"wire_id": 9, "start_s": 763606.331888112, "end_s": 763609.377193393} +{"wire_id": 10, "start_s": 763609.815350681, "end_s": 763612.78998859} +{"wire_id": 11, "start_s": 763613.197026384, "end_s": 763616.141628944} +{"wire_id": 12, "start_s": 763616.431506592, "end_s": 763619.693420484} +{"wire_id": 13, "start_s": 763620.286450803, "end_s": 763623.469771927} +{"wire_id": 14, "start_s": 763624.259804255, "end_s": 763627.425016794} +{"wire_id": 15, "start_s": 763628.021533137, "end_s": 763631.040927777} +{"wire_id": 16, "start_s": 763631.640994818, "end_s": 763634.624162977} +{"wire_id": 17, "start_s": 763635.329653644, "end_s": 763638.41844151} +{"wire_id": 18, "start_s": 763639.009942263, "end_s": 763642.133757822} +{"wire_id": 19, "start_s": 763642.72391619, "end_s": 763645.827582507} +{"wire_id": 20, "start_s": 763646.537111864, "end_s": 763649.684758849} +{"wire_id": 21, "start_s": 763650.278519411, "end_s": 763653.471233431} +{"wire_id": 22, "start_s": 763654.06689781, "end_s": 763657.203362361} +{"wire_id": 23, "start_s": 763657.915828129, "end_s": 763661.0224144} +{"wire_id": 24, "start_s": 763661.621721333, "end_s": 763665.652277616} +{"wire_id": 25, "start_s": 763667.141838262, "end_s": 763670.51644046} +{"wire_id": 26, "start_s": 763672.208959287, "end_s": 763675.755557115} +{"wire_id": 27, "start_s": 763677.216721375, "end_s": 763680.739447264} +{"wire_id": 28, "start_s": 763682.198246191, "end_s": 763685.746247275} +{"wire_id": 29, "start_s": 763687.330901196, "end_s": 763690.861823128} +{"wire_id": 30, "start_s": 763692.326296624, "end_s": 763695.866581528} +{"wire_id": 31, "start_s": 763697.329300985, "end_s": 763700.831297589} +{"wire_id": 32, "start_s": 763702.41706965, "end_s": 763705.962888302} +{"wire_id": 33, "start_s": 763707.422394362, "end_s": 763710.951620017} +{"wire_id": 34, "start_s": 763712.425526863, "end_s": 763715.926748567} +{"wire_id": 35, "start_s": 763717.522637148, "end_s": 763721.154815845} diff --git a/opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen4b-rngd-atom.run.json b/opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen4b-rngd-atom.run.json new file mode 100644 index 00000000..2ca3f429 --- /dev/null +++ b/opt_prime/examples/pd/http_matrix/evidence263/quality263-qwen4b-rngd-atom.run.json @@ -0,0 +1,16 @@ +{ + "stage": "complete", + "route": "rngd-atom", + "model": "Qwen/Qwen3-4B", + "runtime_commit": "3e16c33", + "torch": "2.11.0+cpu", + "tokenizer_inputs_equal": true, + "cases": 36, + "context": 9216, + "batch_size": 1, + "kv_blocks": 3, + "cache": false, + "producer_width": 8192, + "answer_correct": 36, + "format_correct": 36 +} diff --git a/opt_prime/examples/pd/http_matrix/evidence263/quality263-summary.json b/opt_prime/examples/pd/http_matrix/evidence263/quality263-summary.json index c79437bb..d7020eb7 100644 --- a/opt_prime/examples/pd/http_matrix/evidence263/quality263-summary.json +++ b/opt_prime/examples/pd/http_matrix/evidence263/quality263-summary.json @@ -268,7 +268,12 @@ "wrong_cases": [] }, "rngd-atom": { - "stage": "not_collected" + "stage": "complete", + "cases": 36, + "answer_correct": 36, + "format_correct": 36, + "length_limited": 0, + "wrong_cases": [] } }, "pairs": { @@ -325,6 +330,60 @@ "token_equal": 12 } } + }, + "native-gpu→rngd-atom": { + "cases": 36, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 36, + "changes": [], + "by_context": { + "0": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + }, + "2048": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + }, + "8192": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + } + } + }, + "native-atom→rngd-atom": { + "cases": 36, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 36, + "changes": [], + "by_context": { + "0": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + }, + "2048": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + }, + "8192": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + } + } } } }, diff --git a/opt_prime/examples/pd/http_matrix/evidence263/research-events.jsonl b/opt_prime/examples/pd/http_matrix/evidence263/research-events.jsonl index da3753a0..34815188 100644 --- a/opt_prime/examples/pd/http_matrix/evidence263/research-events.jsonl +++ b/opt_prime/examples/pd/http_matrix/evidence263/research-events.jsonl @@ -1,2 +1,8 @@ {"id": "1b0436c1188a4a50b10e9819eae15c8b", "ts": "2026-10-24T05:20:43+09:00", "task": "Structured answer compatibility (aicomp #263)", "result": "Fixed model set includes Qwen3-4B instead of Qwen2.5; preregistered 36 structured cases/model, scorer tests 9 passed after 9 expected failures; hardware inference pending", "next": "Freeze all four tokenizer-specific input sets; finish full suite then run paired GPU/RNGD and ATOM paths", "state": "active", "done": false, "blocker": "", "metrics": {}, "params": {}, "artifacts": [], "experiments": ["experiments/2026-10-24-heldout-quality263.md"], "labels": ["analysis"], "review": false, "todos": [], "note": "", "agent": "codex", "host": "xeon-max", "branch": "main", "commit": "351d39d-dirty"} {"id": "057fd61c56ea49be9b2e62ae9cfae52c", "ts": "2026-10-24T05:23:22+09:00", "task": "Structured answer compatibility (aicomp #263)", "result": "Frozen 144 inputs across four models; 10 evaluator tests pass, full prior collection533passed/15skip/253subtests. Paired nativeGPU/RNGD-GPU queue started after confirming idle hardware.", "next": "Finish 288 GPU-route responses and compare paired correctness, format and tokens; then ATOM routes", "state": "active", "done": false, "blocker": "", "metrics": {}, "params": {}, "artifacts": [{"path": "analysis/serving-matrix-201/quality263-fixture-manifest.json", "size": 1283, "sha256": "326bae03c5f6dea8"}], "experiments": ["experiments/2026-10-24-heldout-quality263.md"], "labels": ["experiment"], "review": false, "todos": [], "note": "", "agent": "codex", "host": "xeon-max", "branch": "main", "commit": "351d39d-dirty"} +{"id": "a9878903b2d141b8bf055f879ebeae3b", "ts": "2026-10-24T05:25:50+09:00", "task": "Structured answer compatibility (aicomp #263)", "result": "PR264 evaluator7b2fd19 published. Qwen0.6 nativeGPU32/36 vs RNGD-GPU34/36 structured answers; regressions0, improvements2, tokens34/36 equal. Other models continue in owned GPU queue PID1285453.", "next": "Finish Qwen3-4B/KONI/Gemma paired GPU runs; extend frozen cases to nativeATOM/RNGD-ATOM without changing prompts", "state": "active", "done": false, "blocker": "", "metrics": {}, "params": {"model": "Qwen3-0.6B", "cases": "36"}, "artifacts": [{"path": "analysis/serving-matrix-201/quality263-qwen06b-gpu-summary.json", "size": 130, "sha256": "3302c86b883d3097"}], "experiments": ["experiments/2026-10-24-heldout-quality263.md"], "labels": ["experiment"], "review": false, "todos": [], "note": "", "agent": "codex", "host": "xeon-max", "branch": "main", "commit": "351d39d-dirty"} +{"id": "d25f1f32226f43059c647d835183fb0e", "ts": "2026-10-24T05:27:38+09:00", "task": "Structured answer compatibility (aicomp #263)", "result": "Qwen4 nativeGPU36/36 structured answers. Owned ATOM Pod started; exact remote tokenizer equality passes frozen36case inputs before inference. GPU PID1285453 and ATOM queuePID1292321 live.", "next": "Collect four-model/four-route heldout responses, paired regressions and immutable input checks", "state": "active", "done": false, "blocker": "", "metrics": {}, "params": {}, "artifacts": [], "experiments": ["experiments/2026-10-24-heldout-quality263.md"], "labels": ["experiment"], "review": false, "todos": [], "note": "", "agent": "codex", "host": "xeon-max", "branch": "main", "commit": "351d39d-dirty"} +{"id": "2ea12a0fbde54700b153763d54620784", "ts": "2026-10-24T05:31:20+09:00", "task": "Structured answer compatibility (aicomp #263)", "result": "Published b3b4a5a to PR264: Qwen4 GPU/RNGD-GPU36/36 answers and36/36 exacttokens; Qwen0.6 pair32vs34 answers. ATOM nativeQwen0.6 complete; liveRNGD-ATOM initialization and KONI GPU comparison still active. Rechecked allfour GPU512-token decode-isolation artifacts passed.", "next": "Collect remaining GPU/ATOM routes from live queues PID1285453/PID1292321; keep strict-token failures separate from structured-answer scores", "state": "active", "done": false, "blocker": "", "metrics": {}, "params": {"model": "Qwen3-4B", "cases": "36"}, "artifacts": [{"path": "analysis/serving-matrix-201/quality263-qwen4b-gpu-summary.json", "size": 130, "sha256": "7cb2aa5aa8d17747"}], "experiments": ["experiments/2026-10-24-heldout-quality263.md"], "labels": ["experiment"], "review": false, "todos": [], "note": "", "agent": "codex", "host": "xeon-max", "branch": "main", "commit": "351d39d-dirty"} +{"id": "d4ffad67020746a0b4b57be8e175f205", "ts": "2026-10-24T05:38:32+09:00", "task": "Structured answer compatibility (aicomp #263)", "result": "All288 GPU responses complete; exacttoken pairs Q06=34/36,Q4=36/36,KONI=35/36,Gemma=32/36. Q06ATOM pair36/36 token equal; Q4nativeATOM36/36 structured score. All21 source checkpoint/config/tokenizer files SHA256-identical across hosts.", "next": "Finish five remaining ATOM routes from live owned queuePID1292321; preserve format/content distinction and strict historical token failures", "state": "active", "done": false, "blocker": "", "metrics": {}, "params": {}, "artifacts": [{"path": "analysis/serving-matrix-201/quality263-summary.json", "size": 27683, "sha256": "0cf7034ef6912788"}, {"path": "analysis/serving-matrix-201/quality263-model-identity.json", "size": 766, "sha256": "8a208a90a6cbc8c6"}], "experiments": ["experiments/2026-10-24-heldout-quality263.md"], "labels": ["experiment"], "review": false, "todos": [], "note": "", "agent": "codex", "host": "xeon-max", "branch": "main", "commit": "351d39d-dirty"} +{"id": "ba772c50c9b14714bc71380ceb9980be", "ts": "2026-10-24T05:39:35+09:00", "task": "Structured answer compatibility (aicomp #263)", "result": "Published all GPU pairs and Q06ATOM pair in PR264 d8e6d0a; source21files SHA256 identical. Published audit rerun matches saved summary after correcting manifest filename. Local accelerator jobs exited; owned ATOM queue1292321 remains active.", "next": "Wait for actual Qwen4/KONI/Gemma ATOM completions, collect evidence and audit remaining compatibility requirements", "state": "active", "done": false, "blocker": "", "metrics": {}, "params": {}, "artifacts": [], "experiments": ["experiments/2026-10-24-heldout-quality263.md"], "labels": ["experiment"], "review": false, "todos": [], "note": "", "agent": "codex", "host": "xeon-max", "branch": "main", "commit": "351d39d-dirty"} +{"id": "74ff20284c5f4c01b485d538923b2200", "ts": "2026-10-24T05:54:06+09:00", "task": "Structured answer compatibility (aicomp #263)", "result": "Qwen3-4B allfour routes complete:36/36 answers each, every paired token sequence exact. Remaining ATOM routes KONI/Gemma running in owned queue1292321. Existing long-free-generation issue256 remains open.", "next": "Finish KONI/Gemma ATOM; publish four-model final quality matrix; continue isolated265layer-progress integration", "state": "active", "done": false, "blocker": "", "metrics": {}, "params": {"model": "Qwen3-4B", "cases": "36"}, "artifacts": [{"path": "analysis/serving-matrix-201/quality263-summary.json", "size": 29294, "sha256": "d5f03ba245ebf8b1"}], "experiments": ["experiments/2026-10-24-heldout-quality263.md"], "labels": ["experiment"], "review": false, "todos": [], "note": "", "agent": "codex", "host": "xeon-max", "branch": "main", "commit": "351d39d-dirty"} diff --git a/opt_prime/examples/pd/http_matrix/experiments/2026-10-24-heldout-quality263.md b/opt_prime/examples/pd/http_matrix/experiments/2026-10-24-heldout-quality263.md index 72067287..c725b7f0 100644 --- a/opt_prime/examples/pd/http_matrix/experiments/2026-10-24-heldout-quality263.md +++ b/opt_prime/examples/pd/http_matrix/experiments/2026-10-24-heldout-quality263.md @@ -76,3 +76,5 @@ ATOM partial completion: Qwen0.6 native33/36 and RNGD33/36 structured answers,36 Both hosts' complete source weight shards, config.json, generation_config.json and tokenizer.json were hashed: all21 files (~27.67GB per host) SHA256-identical across four models. Source revisions and individual hashes are retained in quality263-{local,remote}-model-hashes.json and quality263-model-identity.json. This checks checkpoint source identity, not compiled-artifact internals. Auditor summarize_quality263.py verifies fixture-file hashes, case/input-hash correspondence, response fields, scores recomputed from raw text, and paired transitions. Pending routes remain explicitly not_collected/starting; they are not treated as passes. + +Qwen3-4B allfour routes completed: each36/36 structured answers and36/36 format, every paired token sequence identical acrossGPU/RNGD/ATOM. Remaining ATOM routes: KONI and Gemma. This does not erase the prior different long-free-generation fixture failure#256. From 6d6b078e12a2e92e4282af9171ed639eb9e28943 Mon Sep 17 00:00:00 2001 From: gspark Date: Sat, 24 Oct 2026 06:16:18 +0900 Subject: [PATCH 5/7] test: audit all route pairs and retain ATOM quality counterexample --- opt_prime/examples/pd/http_matrix/STATUS.md | 2 + .../quality263-gemma-native-atom.jsonl | 36 + .../quality263-gemma-native-atom.log | 37 + .../quality263-gemma-native-atom.run.json | 16 + .../quality263-koni-native-atom.jsonl | 36 + .../quality263-koni-native-atom.log | 38 + .../quality263-koni-native-atom.run.json | 16 + .../quality263-koni-rngd-atom.jsonl | 36 + .../evidence263/quality263-koni-rngd-atom.log | 44 + .../quality263-koni-rngd-atom.producer.jsonl | 36 + .../quality263-koni-rngd-atom.run.json | 16 + .../evidence263/quality263-summary.json | 854 +++++++++++++++--- .../evidence263/research-events.jsonl | 1 + .../evidence263/summarize_quality263.py | 4 +- .../2026-10-24-heldout-quality263.md | 6 + 15 files changed, 1060 insertions(+), 118 deletions(-) create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-native-atom.jsonl create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-native-atom.log create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-native-atom.run.json create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-native-atom.jsonl create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-native-atom.log create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-native-atom.run.json create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-rngd-atom.jsonl create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-rngd-atom.log create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-rngd-atom.producer.jsonl create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-rngd-atom.run.json diff --git a/opt_prime/examples/pd/http_matrix/STATUS.md b/opt_prime/examples/pd/http_matrix/STATUS.md index 89181459..8440687d 100644 --- a/opt_prime/examples/pd/http_matrix/STATUS.md +++ b/opt_prime/examples/pd/http_matrix/STATUS.md @@ -57,3 +57,5 @@ Allfour live GPU isolation runs pass: active512-token decode survives other-requ #263 all288 GPU responses complete; structured scores native/RNGD:Q06 32/34,Q4 36/36,KONI31/31,Gemma0/0 (code fences, not zero content accuracy). Exacttoken pairs34/36,36/36,35/36,32/36. Q06 ATOM pair33/33 scores and36/36 tokens equal; Q4nativeATOM36/36. All source weight shards/config/tokenizer SHA256 match between hosts. Remaining fiveATOM routes active. #263 Qwen4allfour routes complete:each36/36 structured answers and all36 token sequences equal acrossGPU/RNGD/ATOM. KONI/GemmaATOM pending; historical#256 remains open. + +#263 six-pair audit: Q06 passkey-3-2048 correct on RNGD→GPU but wrong on bothATOM routes; no universal quality-equivalence claim. KONI allfour complete31/36 scores; nativeGPU/nativeATOM andRNGD-GPU/RNGD-ATOM each36/36 exacttokens. OnlyGemma RNGD-ATOM pending. diff --git a/opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-native-atom.jsonl b/opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-native-atom.jsonl new file mode 100644 index 00000000..b5b1ed87 --- /dev/null +++ b/opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-native-atom.jsonl @@ -0,0 +1,36 @@ +{"case": "arithmetic-0-0", "route": "native-atom", "input_sha256": "ca6bce506a59e3ff2ea456d58be66d889b684a8d3c0d8264ea1adf319eba3215", "input_tokens": 38, "expected": "96", "text": "```json\n{\n \"answer\": 96\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 236743, 236819, 236825, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-0-0", "object": "text_completion", "created": 1792789514, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": 96\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 236743, 236819, 236825, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 38, "completion_tokens": 17, "total_tokens": 55}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 0.20122857205569744} +{"case": "passkey-0-0", "route": "native-atom", "input_sha256": "e6af0e26e172ab6739b897c7e5886bbd244a67d68e688c29251f38f0b623c516", "input_tokens": 57, "expected": "34HYL6W8", "text": "```json\n{\n \"answer\": \"34HYL6W8\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 236800, 236812, 32207, 236798, 236825, 236824, 236828, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-passkey-0-0", "object": "text_completion", "created": 1792789514, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"34HYL6W8\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 236800, 236812, 32207, 236798, 236825, 236824, 236828, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 57, "completion_tokens": 23, "total_tokens": 80}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 0.2617191660683602} +{"case": "korean_table-0-0", "route": "native-atom", "input_sha256": "ea4ffcca39f8322f4ad91bec1ebc6bd77d5e11cfa2c10de1e5e0491a0c771c04", "input_tokens": 68, "expected": "춘천", "text": "```json\n{\n \"answer\": \"춘천\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 242349, 239360, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-korean_table-0-0", "object": "text_completion", "created": 1792789514, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"춘천\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 242349, 239360, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 68, "completion_tokens": 18, "total_tokens": 86}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 0.2078818540321663} +{"case": "arithmetic-1-0", "route": "native-atom", "input_sha256": "9f6534dcba2389afa359614d713c5ccd0c632c8a3d737e948f5744236216751c", "input_tokens": 38, "expected": "83", "text": "```json\n{\n \"answer\": 83\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 236743, 236828, 236800, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-1-0", "object": "text_completion", "created": 1792789514, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": 83\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 236743, 236828, 236800, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 38, "completion_tokens": 17, "total_tokens": 55}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 0.19829980400390923} +{"case": "passkey-1-0", "route": "native-atom", "input_sha256": "8110d36233ff6302384e08812339b32a6505ad4f29827c8fcbc3cc5e6f7f8143", "input_tokens": 54, "expected": "CNBKEFR7", "text": "```json\n{\n \"answer\": \"CNBKEFR7\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 9353, 236799, 11998, 11869, 236832, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-passkey-1-0", "object": "text_completion", "created": 1792789515, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"CNBKEFR7\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 9353, 236799, 11998, 11869, 236832, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 54, "completion_tokens": 21, "total_tokens": 75}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 0.2414139130851254} +{"case": "korean_table-1-0", "route": "native-atom", "input_sha256": "e4ca0aa029d8e1569bb761ba0f1d8945c6ff89c30fc5676f2f5a70ebebebded6", "input_tokens": 68, "expected": "목포", "text": "```json\n{\n \"answer\": \"목포\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 239136, 238709, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-korean_table-1-0", "object": "text_completion", "created": 1792789515, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"목포\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 239136, 238709, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 68, "completion_tokens": 18, "total_tokens": 86}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 0.20915404707193375} +{"case": "arithmetic-2-0", "route": "native-atom", "input_sha256": "86c0f5977a407fb0b704b381d799c8ec10e3c96c8c943eb9d09f69f90bcb7337", "input_tokens": 38, "expected": "43", "text": "```json\n{\n \"answer\": 33\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 236743, 236800, 236800, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-2-0", "object": "text_completion", "created": 1792789515, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": 33\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 236743, 236800, 236800, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 38, "completion_tokens": 17, "total_tokens": 55}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 0.19815890293102711} +{"case": "passkey-2-0", "route": "native-atom", "input_sha256": "beaa3fab948d71c3e9b2793b3fe179ce4cf2eeee17923a2f36cac0da494b4e0e", "input_tokens": 53, "expected": "WMLLRJUL", "text": "```json\n{\n \"answer\": \"WMLLRJUL\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 15378, 2182, 136559, 4057, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-passkey-2-0", "object": "text_completion", "created": 1792789515, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"WMLLRJUL\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 15378, 2182, 136559, 4057, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 53, "completion_tokens": 20, "total_tokens": 73}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 0.23161550099030137} +{"case": "korean_table-2-0", "route": "native-atom", "input_sha256": "6c40dd52b87165747a3f02db798f2ae066b85043ed5d362ce32ca37107517bc5", "input_tokens": 68, "expected": "전주", "text": "```json\n{\n \"answer\": \"전주\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 237773, 237754, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-korean_table-2-0", "object": "text_completion", "created": 1792789515, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"전주\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 237773, 237754, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 68, "completion_tokens": 18, "total_tokens": 86}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 0.20851796097122133} +{"case": "arithmetic-3-0", "route": "native-atom", "input_sha256": "fef18b91d4de61b1ee4714defa961ff75523d237e32fd6d1a9aa05105b64a23e", "input_tokens": 38, "expected": "63", "text": "```json\n{\n \"answer\": 63\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 236743, 236825, 236800, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-3-0", "object": "text_completion", "created": 1792789516, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": 63\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 236743, 236825, 236800, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 38, "completion_tokens": 17, "total_tokens": 55}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 0.1981379520148039} +{"case": "passkey-3-0", "route": "native-atom", "input_sha256": "9d26ce509899d1c3ee7d654c2a5b9eef427fe3f062445022f1c12dc1c7a5d433", "input_tokens": 54, "expected": "SBYZEGE7", "text": "```json\n{\n \"answer\": \"SBYZEGE7\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 236773, 19470, 7942, 8809, 236832, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-passkey-3-0", "object": "text_completion", "created": 1792789516, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"SBYZEGE7\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 236773, 19470, 7942, 8809, 236832, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 54, "completion_tokens": 21, "total_tokens": 75}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 0.24013753200415522} +{"case": "korean_table-3-0", "route": "native-atom", "input_sha256": "80e75874d5f77f73b151cd532d78d8e353da8fa375ce6aa622fb4edaa66b2c11", "input_tokens": 68, "expected": "속초", "text": "```json\n{\n \"answer\": \"부산\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 184764, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-korean_table-3-0", "object": "text_completion", "created": 1792789516, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"부산\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 184764, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 68, "completion_tokens": 17, "total_tokens": 85}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 0.19819610309787095} +{"case": "arithmetic-0-2048", "route": "native-atom", "input_sha256": "af4879aef2e6d64f89abce61ba827a7147ce7aa68f131b5059730f6daedfbf89", "input_tokens": 2048, "expected": "96", "text": "```json\n{\n \"answer\": 96\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 236743, 236819, 236825, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-0-2048", "object": "text_completion", "created": 1792789516, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": 96\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 236743, 236819, 236825, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2048, "completion_tokens": 17, "total_tokens": 2065}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 0.38186334806960076} +{"case": "passkey-0-2048", "route": "native-atom", "input_sha256": "ec3792ad17aed14871774cb4bfb238a098a5fbfab84f26321dbaffef4b4bc183", "input_tokens": 2037, "expected": "34HYL6W8", "text": "```json\n{\n \"answer\": \"34HYL6W8\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 236800, 236812, 32207, 236798, 236825, 236824, 236828, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-passkey-0-2048", "object": "text_completion", "created": 1792789517, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"34HYL6W8\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 236800, 236812, 32207, 236798, 236825, 236824, 236828, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2037, "completion_tokens": 23, "total_tokens": 2060}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 0.4327426420059055} +{"case": "korean_table-0-2048", "route": "native-atom", "input_sha256": "3f9e6fea06717095d1a90b92ac31397cb680a716891bcdd1d20b251fb5511eb5", "input_tokens": 2048, "expected": "춘천", "text": "```json\nnull\n```", "tokens": [2717, 3723, 107, 6455, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-korean_table-0-2048", "object": "text_completion", "created": 1792789517, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\nnull\n```", "token_ids": [2717, 3723, 107, 6455, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2048, "completion_tokens": 7, "total_tokens": 2055}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 0.2619548300281167} +{"case": "arithmetic-1-2048", "route": "native-atom", "input_sha256": "c705989bcfc419d3aa14ab0b042b6e1ae5353866cd792b3c7891b76451e8ee5f", "input_tokens": 2048, "expected": "83", "text": "```json\n{\n \"answer\": 83\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 236743, 236828, 236800, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-1-2048", "object": "text_completion", "created": 1792789517, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": 83\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 236743, 236828, 236800, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2048, "completion_tokens": 17, "total_tokens": 2065}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 0.36822352989111096} +{"case": "passkey-1-2048", "route": "native-atom", "input_sha256": "e1fbe20650a58df3fd16efb5b61c1e1d2a45ea5e677d760cea1ad435789ccec6", "input_tokens": 2034, "expected": "CNBKEFR7", "text": "```json\n{\n \"answer\": \"CNBKEFR7\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 9353, 236799, 11998, 11869, 236832, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-passkey-1-2048", "object": "text_completion", "created": 1792789518, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"CNBKEFR7\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 9353, 236799, 11998, 11869, 236832, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2034, "completion_tokens": 21, "total_tokens": 2055}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 0.41048803203739226} +{"case": "korean_table-1-2048", "route": "native-atom", "input_sha256": "d5a80b27cd41a44c089a38a4654dc064496f7e833b906493f407cf49c12879f5", "input_tokens": 2048, "expected": "목포", "text": "```json\n{\n \"answer\": \"목포\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 239136, 238709, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-korean_table-1-2048", "object": "text_completion", "created": 1792789518, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"목포\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 239136, 238709, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2048, "completion_tokens": 18, "total_tokens": 2066}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 0.38205373100936413} +{"case": "arithmetic-2-2048", "route": "native-atom", "input_sha256": "3718106299b7ba6fbcead43904b7e934e114eeb3f6164df3c7fe1db11599a006", "input_tokens": 2048, "expected": "43", "text": "```json\n{\n \"answer\": 33\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 236743, 236800, 236800, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-2-2048", "object": "text_completion", "created": 1792789519, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": 33\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 236743, 236800, 236800, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2048, "completion_tokens": 17, "total_tokens": 2065}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 0.37025855004321784} +{"case": "passkey-2-2048", "route": "native-atom", "input_sha256": "31115153f401db2cb1466d72cf519d69168b22f449778893b0c3acca34cea0fb", "input_tokens": 2048, "expected": "WMLLRJUL", "text": "```json\n{\n \"answer\": \"WMLLRJUL\"\n}\n```\n", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 15378, 2182, 136559, 4057, 236775, 107, 236783, 107, 2717, 107, 106], "finish_reason": "stop", "response": {"id": "quality263-passkey-2-2048", "object": "text_completion", "created": 1792789519, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"WMLLRJUL\"\n}\n```\n", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 15378, 2182, 136559, 4057, 236775, 107, 236783, 107, 2717, 107, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2048, "completion_tokens": 21, "total_tokens": 2069}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 0.41535092901904136} +{"case": "korean_table-2-2048", "route": "native-atom", "input_sha256": "9599db7b28431faf2421bf2180833e5a4aef98483a7254721f1ec83c92fd46f7", "input_tokens": 2048, "expected": "전주", "text": "```json\n{\n \"answer\": \"전주\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 237773, 237754, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-korean_table-2-2048", "object": "text_completion", "created": 1792789519, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"전주\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 237773, 237754, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2048, "completion_tokens": 18, "total_tokens": 2066}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 0.37819920596666634} +{"case": "arithmetic-3-2048", "route": "native-atom", "input_sha256": "251532c0d3e1841162c6c7e2dcc8bd256cc44a258af9daae4aa3d23024810d7c", "input_tokens": 2048, "expected": "63", "text": "```json\n{\n \"answer\": 63\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 236743, 236825, 236800, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-3-2048", "object": "text_completion", "created": 1792789520, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": 63\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 236743, 236825, 236800, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2048, "completion_tokens": 17, "total_tokens": 2065}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 0.37116554903332144} +{"case": "passkey-3-2048", "route": "native-atom", "input_sha256": "1bfc4b6311f887ae68cf363ab09f2e0115e9a5bce4dee802a84068ee11ea73ee", "input_tokens": 2034, "expected": "SBYZEGE7", "text": "```json\n{\n \"answer\": \"SBYZEGE7\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 236773, 19470, 7942, 8809, 236832, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-passkey-3-2048", "object": "text_completion", "created": 1792789520, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"SBYZEGE7\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 236773, 19470, 7942, 8809, 236832, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2034, "completion_tokens": 21, "total_tokens": 2055}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 0.41558859206270427} +{"case": "korean_table-3-2048", "route": "native-atom", "input_sha256": "3f1148302c3113bb35b82b8c90495e402f0fd8a6ee32cc13ab54f2e88a5bc569", "input_tokens": 2048, "expected": "속초", "text": "```json\n{\n \"answer\": \"속초\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 238701, 239378, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-korean_table-3-2048", "object": "text_completion", "created": 1792789521, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"속초\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 238701, 239378, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2048, "completion_tokens": 18, "total_tokens": 2066}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 0.3807854199549183} +{"case": "arithmetic-0-8192", "route": "native-atom", "input_sha256": "8f09d7d4d70782f016ec9530e14fd8992c798993340fd89dc91a1c1dd8f98686", "input_tokens": 8183, "expected": "96", "text": "```json\n{\n \"answer\": \"96\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 236819, 236825, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-0-8192", "object": "text_completion", "created": 1792789522, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"96\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 236819, 236825, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8183, "completion_tokens": 18, "total_tokens": 8201}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 1.006193727022037} +{"case": "passkey-0-8192", "route": "native-atom", "input_sha256": "04b383a0a4fbcb099f7b148e3e949fde72c21f6231bbec3dce68f53027894edf", "input_tokens": 8187, "expected": "34HYL6W8", "text": "```json\n{\n \"answer\": \"34HYL6W8\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 236800, 236812, 32207, 236798, 236825, 236824, 236828, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-passkey-0-8192", "object": "text_completion", "created": 1792789523, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"34HYL6W8\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 236800, 236812, 32207, 236798, 236825, 236824, 236828, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8187, "completion_tokens": 23, "total_tokens": 8210}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 1.0519393640570343} +{"case": "korean_table-0-8192", "route": "native-atom", "input_sha256": "e770191b7695f0d43ce4a12097d1d2ef0ae30f8c39118e7104e4d21047a4d5c4", "input_tokens": 8183, "expected": "춘천", "text": "```json\n{\n \"answer\": \"부산\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 184764, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-korean_table-0-8192", "object": "text_completion", "created": 1792789524, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"부산\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 184764, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8183, "completion_tokens": 17, "total_tokens": 8200}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 0.9891978190280497} +{"case": "arithmetic-1-8192", "route": "native-atom", "input_sha256": "ae1469d2d3a14cf7da2fa13c1f8128325479c8db9c7bb8f6123fbecd93e3e4cf", "input_tokens": 8183, "expected": "83", "text": "```json\n{\"answer\": \"83\"}\n```", "tokens": [2717, 3723, 107, 14937, 14433, 1083, 623, 236828, 236800, 25938, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-1-8192", "object": "text_completion", "created": 1792789525, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\"answer\": \"83\"}\n```", "token_ids": [2717, 3723, 107, 14937, 14433, 1083, 623, 236828, 236800, 25938, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8183, "completion_tokens": 13, "total_tokens": 8196}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 0.9439002689905465} +{"case": "passkey-1-8192", "route": "native-atom", "input_sha256": "e59b6d8def078023ad5694672bb2bbec6994e459bc39c7cfab95eb2a0418008a", "input_tokens": 8184, "expected": "CNBKEFR7", "text": "```json\n{\n \"answer\": \"CNBKEFR7\"\n}\n```\n", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 9353, 236799, 11998, 11869, 236832, 236775, 107, 236783, 107, 2717, 107, 106], "finish_reason": "stop", "response": {"id": "quality263-passkey-1-8192", "object": "text_completion", "created": 1792789526, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"CNBKEFR7\"\n}\n```\n", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 9353, 236799, 11998, 11869, 236832, 236775, 107, 236783, 107, 2717, 107, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8184, "completion_tokens": 22, "total_tokens": 8206}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 1.0462756140623242} +{"case": "korean_table-1-8192", "route": "native-atom", "input_sha256": "9192491b17b6fa3a60efa8f084df443ffe2522a806a7eb6abbf9904ad6a5a50e", "input_tokens": 8183, "expected": "목포", "text": "```json\n{\n \"answer\": \"목포\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 239136, 238709, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-korean_table-1-8192", "object": "text_completion", "created": 1792789527, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"목포\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 239136, 238709, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8183, "completion_tokens": 18, "total_tokens": 8201}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 1.0007178611122072} +{"case": "arithmetic-2-8192", "route": "native-atom", "input_sha256": "04e5f9a4c4bdf5b4da2accc13cc755f3c422b8ff16dc60383234ed0ce0c1d744", "input_tokens": 8183, "expected": "43", "text": "```json\n{\"answer\":\"33\"}\n```", "tokens": [2717, 3723, 107, 14937, 14433, 12375, 236800, 236800, 25938, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-2-8192", "object": "text_completion", "created": 1792789528, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\"answer\":\"33\"}\n```", "token_ids": [2717, 3723, 107, 14937, 14433, 12375, 236800, 236800, 25938, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8183, "completion_tokens": 12, "total_tokens": 8195}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 0.9364523710682988} +{"case": "passkey-2-8192", "route": "native-atom", "input_sha256": "bf1531f20519b1d1162a4230d9835ed8e395f12c9c94444744271828d7dea36a", "input_tokens": 8183, "expected": "WMLLRJUL", "text": "```json\n{\n \"answer\": \"WMLLRJUL\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 15378, 2182, 136559, 4057, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-passkey-2-8192", "object": "text_completion", "created": 1792789529, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"WMLLRJUL\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 15378, 2182, 136559, 4057, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8183, "completion_tokens": 20, "total_tokens": 8203}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 1.0230182459345087} +{"case": "korean_table-2-8192", "route": "native-atom", "input_sha256": "bc8f317a6ec4db698ed1f794e9d056bc62a289ce6f49ff08b918454cf800456e", "input_tokens": 8183, "expected": "전주", "text": "```json\n{\n \"answer\": \"전주\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 237773, 237754, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-korean_table-2-8192", "object": "text_completion", "created": 1792789530, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"전주\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 237773, 237754, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8183, "completion_tokens": 18, "total_tokens": 8201}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 0.9982714019715786} +{"case": "arithmetic-3-8192", "route": "native-atom", "input_sha256": "a1fb40e56504a08b4fea4da371f0fce7183ca271335fa8039921ecf9f752868c", "input_tokens": 8183, "expected": "63", "text": "```json\n{\"answer\": \"63\"}\n```", "tokens": [2717, 3723, 107, 14937, 14433, 1083, 623, 236825, 236800, 25938, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-3-8192", "object": "text_completion", "created": 1792789531, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\"answer\": \"63\"}\n```", "token_ids": [2717, 3723, 107, 14937, 14433, 1083, 623, 236825, 236800, 25938, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8183, "completion_tokens": 13, "total_tokens": 8196}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 0.9480245650047436} +{"case": "passkey-3-8192", "route": "native-atom", "input_sha256": "f3ba91ce9a3a993574e925264bc3207381e9bfb0710fbf6a2fa3f424774268b1", "input_tokens": 8184, "expected": "SBYZEGE7", "text": "```json\n{\n \"answer\": \"SBYZEGE7\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 236773, 19470, 7942, 8809, 236832, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-passkey-3-8192", "object": "text_completion", "created": 1792789532, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"SBYZEGE7\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 236773, 19470, 7942, 8809, 236832, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8184, "completion_tokens": 21, "total_tokens": 8205}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 1.027090738993138} +{"case": "korean_table-3-8192", "route": "native-atom", "input_sha256": "4ef06a4874f17e50cec21d5f2611a9ff97e8c19b35a1446fa0cafd1f57c3fb40", "input_tokens": 8183, "expected": "속초", "text": "```json\n{\n \"answer\": \"부산\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 184764, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-korean_table-3-8192", "object": "text_completion", "created": 1792789533, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"부산\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 184764, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8183, "completion_tokens": 17, "total_tokens": 8200}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 0.9868242860538885} diff --git a/opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-native-atom.log b/opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-native-atom.log new file mode 100644 index 00000000..e287ebe7 --- /dev/null +++ b/opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-native-atom.log @@ -0,0 +1,37 @@ +ATOM_BINDING {"visible_count": 1, "physical": 1, "opened": ["/dev/rbln1"]} +arithmetic-0-0 False False +passkey-0-0 False False +korean_table-0-0 False False +arithmetic-1-0 False False +passkey-1-0 False False +korean_table-1-0 False False +arithmetic-2-0 False False +passkey-2-0 False False +korean_table-2-0 False False +arithmetic-3-0 False False +passkey-3-0 False False +korean_table-3-0 False False +arithmetic-0-2048 False False +passkey-0-2048 False False +korean_table-0-2048 False False +arithmetic-1-2048 False False +passkey-1-2048 False False +korean_table-1-2048 False False +arithmetic-2-2048 False False +passkey-2-2048 False False +korean_table-2-2048 False False +arithmetic-3-2048 False False +passkey-3-2048 False False +korean_table-3-2048 False False +arithmetic-0-8192 False False +passkey-0-8192 False False +korean_table-0-8192 False False +arithmetic-1-8192 False False +passkey-1-8192 False False +korean_table-1-8192 False False +arithmetic-2-8192 False False +passkey-2-8192 False False +korean_table-2-8192 False False +arithmetic-3-8192 False False +passkey-3-8192 False False +korean_table-3-8192 False False diff --git a/opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-native-atom.run.json b/opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-native-atom.run.json new file mode 100644 index 00000000..f5f041c3 --- /dev/null +++ b/opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-native-atom.run.json @@ -0,0 +1,16 @@ +{ + "stage": "complete", + "route": "native-atom", + "model": "unsloth/gemma-3-1b-it", + "runtime_commit": "3e16c33", + "torch": "2.11.0+cpu", + "tokenizer_inputs_equal": true, + "cases": 36, + "context": 9216, + "batch_size": 1, + "kv_blocks": 3, + "cache": false, + "producer_width": 8192, + "answer_correct": 0, + "format_correct": 0 +} diff --git a/opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-native-atom.jsonl b/opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-native-atom.jsonl new file mode 100644 index 00000000..f5d2e9c3 --- /dev/null +++ b/opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-native-atom.jsonl @@ -0,0 +1,36 @@ +{"case": "arithmetic-0-0", "route": "native-atom", "input_sha256": "0d5cbdfc04d8d8d4abc3243f4d3df37dac0664f1c9a910f6789cce0e8c022d6f", "input_tokens": 150, "expected": "96", "text": "{\n \"answer\": \"96\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 4161, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-0-0", "object": "text_completion", "created": 1792788772, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"96\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 4161, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 150, "completion_tokens": 10, "total_tokens": 160}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 0.8063394429627806} +{"case": "passkey-0-0", "route": "native-atom", "input_sha256": "148a205e1ed4cf9dea9bf1a236ed8d55273022522455de9786990af33647db4f", "input_tokens": 169, "expected": "34HYL6W8", "text": "{\n \"answer\": \"34HYL6W8\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 1958, 49952, 43, 21, 54, 23, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-passkey-0-0", "object": "text_completion", "created": 1792788773, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"34HYL6W8\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 1958, 49952, 43, 21, 54, 23, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 169, "completion_tokens": 15, "total_tokens": 184}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 1.1421866780146956} +{"case": "korean_table-0-0", "route": "native-atom", "input_sha256": "f7a96c6c61f14373ba3ecd21a228d373702ffd016002074ea5c76b914e787d9b", "input_tokens": 182, "expected": "춘천", "text": "{\"answer\": \"부산\"}", "tokens": [5018, 9399, 794, 330, 64189, 86157, 9388, 128009], "finish_reason": "stop", "response": {"id": "quality263-korean_table-0-0", "object": "text_completion", "created": 1792788773, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\"answer\": \"부산\"}", "token_ids": [5018, 9399, 794, 330, 64189, 86157, 9388, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 182, "completion_tokens": 8, "total_tokens": 190}}, "answer_correct": false, "format_correct": true, "elapsed_seconds": 0.6708941559772938} +{"case": "arithmetic-1-0", "route": "native-atom", "input_sha256": "de07f2ba6e0e0c54d00299bec55bfea0bca3115535a319781b982243ed30bc72", "input_tokens": 150, "expected": "83", "text": "{\n \"answer\": \"48 + 35 = 83\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 2166, 489, 220, 1758, 284, 220, 6069, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-1-0", "object": "text_completion", "created": 1792788775, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"48 + 35 = 83\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 2166, 489, 220, 1758, 284, 220, 6069, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 150, "completion_tokens": 16, "total_tokens": 166}}, "answer_correct": false, "format_correct": true, "elapsed_seconds": 1.2093586371047422} +{"case": "passkey-1-0", "route": "native-atom", "input_sha256": "a19a5cd366d8c5db6307c579eacddc7c523b61a88765a4b496acf7421c5f0977", "input_tokens": 167, "expected": "CNBKEFR7", "text": "{\n \"answer\": \"CNBKEFR7\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 29768, 33, 3472, 10725, 22, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-passkey-1-0", "object": "text_completion", "created": 1792788776, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"CNBKEFR7\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 29768, 33, 3472, 10725, 22, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 167, "completion_tokens": 14, "total_tokens": 181}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 1.074304271955043} +{"case": "korean_table-1-0", "route": "native-atom", "input_sha256": "46904506fbff1f58eb3caa030a441988b3fd92df83c12ebddf6ed596cd07c648", "input_tokens": 182, "expected": "목포", "text": "{\"answer\": \"목포\"}", "tokens": [5018, 9399, 794, 330, 88708, 101796, 9388, 128009], "finish_reason": "stop", "response": {"id": "quality263-korean_table-1-0", "object": "text_completion", "created": 1792788776, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\"answer\": \"목포\"}", "token_ids": [5018, 9399, 794, 330, 88708, 101796, 9388, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 182, "completion_tokens": 8, "total_tokens": 190}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 0.6741422610357404} +{"case": "arithmetic-2-0", "route": "native-atom", "input_sha256": "5d57407e124bd1b645bb44e6156a7e8b8774dbf31049daf0aefdf36aba6ffabf", "input_tokens": 150, "expected": "43", "text": "{\n \"answer\": \"The sum of 32 and 11 is 43.\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 791, 2694, 315, 220, 843, 323, 220, 806, 374, 220, 3391, 10246, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-2-0", "object": "text_completion", "created": 1792788778, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"The sum of 32 and 11 is 43.\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 791, 2694, 315, 220, 843, 323, 220, 806, 374, 220, 3391, 10246, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 150, "completion_tokens": 20, "total_tokens": 170}}, "answer_correct": false, "format_correct": true, "elapsed_seconds": 1.4772264510393143} +{"case": "passkey-2-0", "route": "native-atom", "input_sha256": "67df48736d5f8eed8ed87cb21f9bbf14515b4a30c7002f04e8a029e188dea2fd", "input_tokens": 167, "expected": "WMLLRJUL", "text": "{\n \"answer\": \"WMLLRJUL\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 54, 2735, 20721, 41, 1112, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-passkey-2-0", "object": "text_completion", "created": 1792788779, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"WMLLRJUL\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 54, 2735, 20721, 41, 1112, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 167, "completion_tokens": 14, "total_tokens": 181}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 1.074556282022968} +{"case": "korean_table-2-0", "route": "native-atom", "input_sha256": "603c9dece7072b82f4958eaffa237fe469ca506451e0a250bee5530a652beb51", "input_tokens": 182, "expected": "전주", "text": "{\"answer\": \"전주\"}", "tokens": [5018, 9399, 794, 330, 66965, 55430, 9388, 128009], "finish_reason": "stop", "response": {"id": "quality263-korean_table-2-0", "object": "text_completion", "created": 1792788780, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\"answer\": \"전주\"}", "token_ids": [5018, 9399, 794, 330, 66965, 55430, 9388, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 182, "completion_tokens": 8, "total_tokens": 190}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 0.6719808459747583} +{"case": "arithmetic-3-0", "route": "native-atom", "input_sha256": "163a31cf837d8a76b757bcfd787ed0639cca12c434e8528993bddd4c7966338f", "input_tokens": 150, "expected": "63", "text": "{\n \"answer\": \"63\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 5495, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-3-0", "object": "text_completion", "created": 1792788780, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"63\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 5495, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 150, "completion_tokens": 10, "total_tokens": 160}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 0.8066368539584801} +{"case": "passkey-3-0", "route": "native-atom", "input_sha256": "99e6bcbba020691ee22c29fbdc8eb6c89359c3e8647f833586a2805ba089cc9a", "input_tokens": 168, "expected": "SBYZEGE7", "text": "{\n \"answer\": \"SBYZEGE7\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 17094, 41309, 9560, 36, 22, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-passkey-3-0", "object": "text_completion", "created": 1792788781, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"SBYZEGE7\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 17094, 41309, 9560, 36, 22, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 168, "completion_tokens": 14, "total_tokens": 182}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 1.0747308069840074} +{"case": "korean_table-3-0", "route": "native-atom", "input_sha256": "80c3a0a96f8fc932afa2501e8fc3bd7d7c0a57b3d22c68a0fd8b20c7a1434527", "input_tokens": 182, "expected": "속초", "text": "{\"answer\": \"속초\"}", "tokens": [5018, 9399, 794, 330, 102130, 103719, 9388, 128009], "finish_reason": "stop", "response": {"id": "quality263-korean_table-3-0", "object": "text_completion", "created": 1792788782, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\"answer\": \"속초\"}", "token_ids": [5018, 9399, 794, 330, 102130, 103719, 9388, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 182, "completion_tokens": 8, "total_tokens": 190}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 0.6722363679436967} +{"case": "arithmetic-0-2048", "route": "native-atom", "input_sha256": "d5e3fc2d079f487aa54bf12711c278e030ae54da55f6cb57570ce489730d1260", "input_tokens": 2040, "expected": "96", "text": "{\n \"answer\": \"96\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 4161, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-0-2048", "object": "text_completion", "created": 1792788784, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"96\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 4161, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2040, "completion_tokens": 10, "total_tokens": 2050}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 2.243248249986209} +{"case": "passkey-0-2048", "route": "native-atom", "input_sha256": "6cb719c60a8d55b8ad05fa80978bbf003d37f93c9564b7425493cd5c951facfc", "input_tokens": 2045, "expected": "34HYL6W8", "text": "{\n \"answer\": \"34HYL6W8\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 1958, 49952, 43, 21, 54, 23, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-passkey-0-2048", "object": "text_completion", "created": 1792788787, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"34HYL6W8\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 1958, 49952, 43, 21, 54, 23, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2045, "completion_tokens": 15, "total_tokens": 2060}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 2.5805677130119875} +{"case": "korean_table-0-2048", "route": "native-atom", "input_sha256": "2de6ba8e4aab29c2b729c718bbee240e5b5b5f704d658667fb633ccff3950cdc", "input_tokens": 2044, "expected": "춘천", "text": "{\n \"answer\": \"춘천\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 122432, 101584, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-korean_table-0-2048", "object": "text_completion", "created": 1792788789, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"춘천\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 122432, 101584, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2044, "completion_tokens": 11, "total_tokens": 2055}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 2.3120936829363927} +{"case": "arithmetic-1-2048", "route": "native-atom", "input_sha256": "ad0b2916c8ea5d6bbba7adeb75c78be114d5961607cf11fd4876214fe76f2cb8", "input_tokens": 2040, "expected": "83", "text": "{\n \"answer\": \"83\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 6069, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-1-2048", "object": "text_completion", "created": 1792788791, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"83\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 6069, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2040, "completion_tokens": 10, "total_tokens": 2050}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 2.2428762119961902} +{"case": "passkey-1-2048", "route": "native-atom", "input_sha256": "c6f1f8c084a0f7adf16c51f2b1914dfb4da2d68c8c3a245849e38b9837183118", "input_tokens": 2043, "expected": "CNBKEFR7", "text": "{\n \"answer\": \"CNBKEFR7\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 29768, 33, 3472, 10725, 22, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-passkey-1-2048", "object": "text_completion", "created": 1792788794, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"CNBKEFR7\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 29768, 33, 3472, 10725, 22, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2043, "completion_tokens": 14, "total_tokens": 2057}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 2.516404736088589} +{"case": "korean_table-1-2048", "route": "native-atom", "input_sha256": "a60cd701a8ef4007962836a35beef128c02ee2985b13b7fcbe1d21e7849ea514", "input_tokens": 2044, "expected": "목포", "text": "{\n \"answer\": \"목포\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 88708, 101796, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-korean_table-1-2048", "object": "text_completion", "created": 1792788796, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"목포\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 88708, 101796, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2044, "completion_tokens": 11, "total_tokens": 2055}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 2.3110244090203196} +{"case": "arithmetic-2-2048", "route": "native-atom", "input_sha256": "5f0f4897424086c68ec9f858d7c2179b0335fded13c090f716857d5f9ba0bc41", "input_tokens": 2040, "expected": "43", "text": "{\n \"answer\": \"43\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 3391, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-2-2048", "object": "text_completion", "created": 1792788799, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"43\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 3391, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2040, "completion_tokens": 10, "total_tokens": 2050}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 2.2431005319813266} +{"case": "passkey-2-2048", "route": "native-atom", "input_sha256": "793cc606a21f2d953fec07a91e099d20f3624409040251105573dab8236dad2e", "input_tokens": 2043, "expected": "WMLLRJUL", "text": "{\n \"answer\": \"WMLLRJUL\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 54, 2735, 20721, 41, 1112, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-passkey-2-2048", "object": "text_completion", "created": 1792788801, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"WMLLRJUL\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 54, 2735, 20721, 41, 1112, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2043, "completion_tokens": 14, "total_tokens": 2057}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 2.5119675419991836} +{"case": "korean_table-2-2048", "route": "native-atom", "input_sha256": "adec171bcbf4600c8d59467a432af029bc475ce9575c1f4817eece553e37a027", "input_tokens": 2044, "expected": "전주", "text": "{\n \"answer\": \"전주\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 66965, 55430, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-korean_table-2-2048", "object": "text_completion", "created": 1792788803, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"전주\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 66965, 55430, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2044, "completion_tokens": 11, "total_tokens": 2055}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 2.3103487459011376} +{"case": "arithmetic-3-2048", "route": "native-atom", "input_sha256": "679e19177db7acb1a7dd097a45e30f3e005511311577f7e9e9add8b11c6e9933", "input_tokens": 2040, "expected": "63", "text": "{\n \"answer\": \"63\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 5495, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-3-2048", "object": "text_completion", "created": 1792788806, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"63\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 5495, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2040, "completion_tokens": 10, "total_tokens": 2050}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 2.2475822180276737} +{"case": "passkey-3-2048", "route": "native-atom", "input_sha256": "f4347eded9d6462727022c7ab97a9f961d43f4c44828ff7cf901fcd2a7e05510", "input_tokens": 2044, "expected": "SBYZEGE7", "text": "{\n \"answer\": \"SBYZEGE7\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 17094, 41309, 9560, 36, 22, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-passkey-3-2048", "object": "text_completion", "created": 1792788808, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"SBYZEGE7\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 17094, 41309, 9560, 36, 22, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2044, "completion_tokens": 14, "total_tokens": 2058}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 2.512888388009742} +{"case": "korean_table-3-2048", "route": "native-atom", "input_sha256": "6d84fc95ed8b4b55fda6af31a18bb7c32b6f19ab02d745fddd56f5ae5d74efb4", "input_tokens": 2044, "expected": "속초", "text": "{\n \"answer\": \"속초\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 102130, 103719, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-korean_table-3-2048", "object": "text_completion", "created": 1792788810, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"속초\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 102130, 103719, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2044, "completion_tokens": 11, "total_tokens": 2055}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 2.3106573990080506} +{"case": "arithmetic-0-8192", "route": "native-atom", "input_sha256": "b5ac979ed58c0020fb09bed6e865da33798778a4e3f94b9bbd5608832c418e7b", "input_tokens": 8186, "expected": "96", "text": "{\n \"answer\": \"96\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 4161, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-0-8192", "object": "text_completion", "created": 1792788818, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"96\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 4161, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8186, "completion_tokens": 10, "total_tokens": 8196}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 7.9506545721087605} +{"case": "passkey-0-8192", "route": "native-atom", "input_sha256": "d5cd6c30057ebdf9e65495c4a7bd757f811bf4e64860188c7357a67414f94222", "input_tokens": 8191, "expected": "34HYL6W8", "text": "34HYL6W8", "tokens": [1958, 49952, 43, 21, 54, 23, 128009], "finish_reason": "stop", "response": {"id": "quality263-passkey-0-8192", "object": "text_completion", "created": 1792788826, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "34HYL6W8", "token_ids": [1958, 49952, 43, 21, 54, 23, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8191, "completion_tokens": 7, "total_tokens": 8198}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 7.733989814994857} +{"case": "korean_table-0-8192", "route": "native-atom", "input_sha256": "cb0f1c7ae26517f1e9d859e6d0d24aed6e4dc2cd4d3db9fb4983c5a01d71f745", "input_tokens": 8190, "expected": "춘천", "text": "{\n \"answer\": \"춘천\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 122432, 101584, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-korean_table-0-8192", "object": "text_completion", "created": 1792788834, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"춘천\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 122432, 101584, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8190, "completion_tokens": 11, "total_tokens": 8201}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 8.020284185069613} +{"case": "arithmetic-1-8192", "route": "native-atom", "input_sha256": "41e87fb89c71cd493cec4801e39fccc4f625e205a485f48b1310880c5e72173f", "input_tokens": 8186, "expected": "83", "text": "{\n \"answer\": \"83\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 6069, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-1-8192", "object": "text_completion", "created": 1792788842, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"83\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 6069, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8186, "completion_tokens": 10, "total_tokens": 8196}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 7.948859789059497} +{"case": "passkey-1-8192", "route": "native-atom", "input_sha256": "6aec4ccde3c8e068a525f3141930fd0dd89b0066fb57ae44bf1f3695d03a9d38", "input_tokens": 8189, "expected": "CNBKEFR7", "text": "{\n \"answer\": \"CNBKEFR7\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 29768, 33, 3472, 10725, 22, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-passkey-1-8192", "object": "text_completion", "created": 1792788850, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"CNBKEFR7\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 29768, 33, 3472, 10725, 22, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8189, "completion_tokens": 14, "total_tokens": 8203}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 8.235838730004616} +{"case": "korean_table-1-8192", "route": "native-atom", "input_sha256": "a45e928bc070aa02a52f8f8c09fe63eb2c240804cb082b8d38395f45f19e2ac7", "input_tokens": 8190, "expected": "목포", "text": "{\n \"answer\": \"목포\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 88708, 101796, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-korean_table-1-8192", "object": "text_completion", "created": 1792788858, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"목포\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 88708, 101796, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8190, "completion_tokens": 11, "total_tokens": 8201}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 8.021923388005234} +{"case": "arithmetic-2-8192", "route": "native-atom", "input_sha256": "3b2113bbe36b0528ec715ac215ad332254fdcc741b55bba2fad14beb172ce03c", "input_tokens": 8186, "expected": "43", "text": "{\n \"answer\": \"43\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 3391, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-2-8192", "object": "text_completion", "created": 1792788866, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"43\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 3391, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8186, "completion_tokens": 10, "total_tokens": 8196}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 7.9510668410221115} +{"case": "passkey-2-8192", "route": "native-atom", "input_sha256": "e8784fdb10cbfc3bc05540d0a7c9cad53c915638b3887a205ea7df911b53db53", "input_tokens": 8189, "expected": "WMLLRJUL", "text": "WMLLRJUL", "tokens": [54, 2735, 20721, 41, 1112, 128009], "finish_reason": "stop", "response": {"id": "quality263-passkey-2-8192", "object": "text_completion", "created": 1792788874, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "WMLLRJUL", "token_ids": [54, 2735, 20721, 41, 1112, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8189, "completion_tokens": 6, "total_tokens": 8195}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 7.665244259987958} +{"case": "korean_table-2-8192", "route": "native-atom", "input_sha256": "e2a1f3ecbfbde65119093f3e14d982ffe15fc04ff0f46e6dfb0a73aafb6bc301", "input_tokens": 8190, "expected": "전주", "text": "{\n \"answer\": \"전주\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 66965, 55430, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-korean_table-2-8192", "object": "text_completion", "created": 1792788882, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"전주\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 66965, 55430, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8190, "completion_tokens": 11, "total_tokens": 8201}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 8.025514162960462} +{"case": "arithmetic-3-8192", "route": "native-atom", "input_sha256": "9649667fd3e0e3f487ddf07287c5f43d01c9d0a4545f6fb163c215464c0ba727", "input_tokens": 8186, "expected": "63", "text": "{\n \"answer\": \"63\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 5495, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-3-8192", "object": "text_completion", "created": 1792788890, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"63\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 5495, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8186, "completion_tokens": 10, "total_tokens": 8196}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 7.951001578941941} +{"case": "passkey-3-8192", "route": "native-atom", "input_sha256": "e2301ac0d434cc1b0872c0ae4240fa21c7a89cda9a5727ff7adfbb66a6757fb3", "input_tokens": 8190, "expected": "SBYZEGE7", "text": "{\n \"answer\": \"SBYZEGE7\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 17094, 41309, 9560, 36, 22, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-passkey-3-8192", "object": "text_completion", "created": 1792788898, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"SBYZEGE7\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 17094, 41309, 9560, 36, 22, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8190, "completion_tokens": 14, "total_tokens": 8204}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 8.236702271969989} +{"case": "korean_table-3-8192", "route": "native-atom", "input_sha256": "ddf60d0902dc767a5b12fa8c67ed1c79a410df9d76a63d7fcccd2f21b5807ce7", "input_tokens": 8190, "expected": "속초", "text": "{\n \"answer\": \"속초\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 102130, 103719, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-korean_table-3-8192", "object": "text_completion", "created": 1792788906, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"속초\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 102130, 103719, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8190, "completion_tokens": 11, "total_tokens": 8201}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 8.019978774012998} diff --git a/opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-native-atom.log b/opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-native-atom.log new file mode 100644 index 00000000..71c15b62 --- /dev/null +++ b/opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-native-atom.log @@ -0,0 +1,38 @@ +ATOM_BINDING {"visible_count": 1, "physical": 1, "opened": ["/dev/rbln1"]} +[transformers] Ignoring clean_up_tokenization_spaces=True for BPE tokenizer TokenizersBackend. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway. +arithmetic-0-0 True True +passkey-0-0 True True +korean_table-0-0 False True +arithmetic-1-0 False True +passkey-1-0 True True +korean_table-1-0 True True +arithmetic-2-0 False True +passkey-2-0 True True +korean_table-2-0 True True +arithmetic-3-0 True True +passkey-3-0 True True +korean_table-3-0 True True +arithmetic-0-2048 True True +passkey-0-2048 True True +korean_table-0-2048 True True +arithmetic-1-2048 True True +passkey-1-2048 True True +korean_table-1-2048 True True +arithmetic-2-2048 True True +passkey-2-2048 True True +korean_table-2-2048 True True +arithmetic-3-2048 True True +passkey-3-2048 True True +korean_table-3-2048 True True +arithmetic-0-8192 True True +passkey-0-8192 False False +korean_table-0-8192 True True +arithmetic-1-8192 True True +passkey-1-8192 True True +korean_table-1-8192 True True +arithmetic-2-8192 True True +passkey-2-8192 False False +korean_table-2-8192 True True +arithmetic-3-8192 True True +passkey-3-8192 True True +korean_table-3-8192 True True diff --git a/opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-native-atom.run.json b/opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-native-atom.run.json new file mode 100644 index 00000000..cf276b88 --- /dev/null +++ b/opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-native-atom.run.json @@ -0,0 +1,16 @@ +{ + "stage": "complete", + "route": "native-atom", + "model": "KISTI-KONI/KONI-Llama3.1-8B-Instruct-20241024", + "runtime_commit": "3e16c33", + "torch": "2.11.0+cpu", + "tokenizer_inputs_equal": true, + "cases": 36, + "context": 9216, + "batch_size": 1, + "kv_blocks": 1, + "cache": false, + "producer_width": 8192, + "answer_correct": 31, + "format_correct": 34 +} diff --git a/opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-rngd-atom.jsonl b/opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-rngd-atom.jsonl new file mode 100644 index 00000000..cc7b669a --- /dev/null +++ b/opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-rngd-atom.jsonl @@ -0,0 +1,36 @@ +{"case": "arithmetic-0-0", "route": "rngd-atom", "input_sha256": "0d5cbdfc04d8d8d4abc3243f4d3df37dac0664f1c9a910f6789cce0e8c022d6f", "input_tokens": 150, "expected": "96", "text": "{\n \"answer\": \"96\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 4161, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-0-0", "object": "text_completion", "created": 1792789339, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"96\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 4161, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 150, "completion_tokens": 10, "total_tokens": 160}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 4.2400812699925154} +{"case": "passkey-0-0", "route": "rngd-atom", "input_sha256": "148a205e1ed4cf9dea9bf1a236ed8d55273022522455de9786990af33647db4f", "input_tokens": 169, "expected": "34HYL6W8", "text": "{\n \"answer\": \"34HYL6W8\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 1958, 49952, 43, 21, 54, 23, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-passkey-0-0", "object": "text_completion", "created": 1792789343, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"34HYL6W8\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 1958, 49952, 43, 21, 54, 23, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 169, "completion_tokens": 15, "total_tokens": 184}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 4.170760320033878} +{"case": "korean_table-0-0", "route": "rngd-atom", "input_sha256": "f7a96c6c61f14373ba3ecd21a228d373702ffd016002074ea5c76b914e787d9b", "input_tokens": 182, "expected": "춘천", "text": "{\"answer\": \"부산\"}", "tokens": [5018, 9399, 794, 330, 64189, 86157, 9388, 128009], "finish_reason": "stop", "response": {"id": "quality263-korean_table-0-0", "object": "text_completion", "created": 1792789347, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\"answer\": \"부산\"}", "token_ids": [5018, 9399, 794, 330, 64189, 86157, 9388, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 182, "completion_tokens": 8, "total_tokens": 190}}, "answer_correct": false, "format_correct": true, "elapsed_seconds": 3.717238503973931} +{"case": "arithmetic-1-0", "route": "rngd-atom", "input_sha256": "de07f2ba6e0e0c54d00299bec55bfea0bca3115535a319781b982243ed30bc72", "input_tokens": 150, "expected": "83", "text": "{\n \"answer\": \"48 + 35 = 83\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 2166, 489, 220, 1758, 284, 220, 6069, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-1-0", "object": "text_completion", "created": 1792789351, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"48 + 35 = 83\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 2166, 489, 220, 1758, 284, 220, 6069, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 150, "completion_tokens": 16, "total_tokens": 166}}, "answer_correct": false, "format_correct": true, "elapsed_seconds": 4.177632560953498} +{"case": "passkey-1-0", "route": "rngd-atom", "input_sha256": "a19a5cd366d8c5db6307c579eacddc7c523b61a88765a4b496acf7421c5f0977", "input_tokens": 167, "expected": "CNBKEFR7", "text": "{\n \"answer\": \"CNBKEFR7\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 29768, 33, 3472, 10725, 22, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-passkey-1-0", "object": "text_completion", "created": 1792789355, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"CNBKEFR7\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 29768, 33, 3472, 10725, 22, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 167, "completion_tokens": 14, "total_tokens": 181}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 4.074169632047415} +{"case": "korean_table-1-0", "route": "rngd-atom", "input_sha256": "46904506fbff1f58eb3caa030a441988b3fd92df83c12ebddf6ed596cd07c648", "input_tokens": 182, "expected": "목포", "text": "{\"answer\": \"목포\"}", "tokens": [5018, 9399, 794, 330, 88708, 101796, 9388, 128009], "finish_reason": "stop", "response": {"id": "quality263-korean_table-1-0", "object": "text_completion", "created": 1792789359, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\"answer\": \"목포\"}", "token_ids": [5018, 9399, 794, 330, 88708, 101796, 9388, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 182, "completion_tokens": 8, "total_tokens": 190}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 3.685215792967938} +{"case": "arithmetic-2-0", "route": "rngd-atom", "input_sha256": "5d57407e124bd1b645bb44e6156a7e8b8774dbf31049daf0aefdf36aba6ffabf", "input_tokens": 150, "expected": "43", "text": "{\n \"answer\": \"The sum of 32 and 11 is 43.\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 791, 2694, 315, 220, 843, 323, 220, 806, 374, 220, 3391, 10246, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-2-0", "object": "text_completion", "created": 1792789363, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"The sum of 32 and 11 is 43.\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 791, 2694, 315, 220, 843, 323, 220, 806, 374, 220, 3391, 10246, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 150, "completion_tokens": 20, "total_tokens": 170}}, "answer_correct": false, "format_correct": true, "elapsed_seconds": 4.46257491002325} +{"case": "passkey-2-0", "route": "rngd-atom", "input_sha256": "67df48736d5f8eed8ed87cb21f9bbf14515b4a30c7002f04e8a029e188dea2fd", "input_tokens": 167, "expected": "WMLLRJUL", "text": "{\n \"answer\": \"WMLLRJUL\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 54, 2735, 20721, 41, 1112, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-passkey-2-0", "object": "text_completion", "created": 1792789367, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"WMLLRJUL\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 54, 2735, 20721, 41, 1112, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 167, "completion_tokens": 14, "total_tokens": 181}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 4.113578801974654} +{"case": "korean_table-2-0", "route": "rngd-atom", "input_sha256": "603c9dece7072b82f4958eaffa237fe469ca506451e0a250bee5530a652beb51", "input_tokens": 182, "expected": "전주", "text": "{\"answer\": \"전주\"}", "tokens": [5018, 9399, 794, 330, 66965, 55430, 9388, 128009], "finish_reason": "stop", "response": {"id": "quality263-korean_table-2-0", "object": "text_completion", "created": 1792789371, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\"answer\": \"전주\"}", "token_ids": [5018, 9399, 794, 330, 66965, 55430, 9388, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 182, "completion_tokens": 8, "total_tokens": 190}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 3.6530155099462718} +{"case": "arithmetic-3-0", "route": "rngd-atom", "input_sha256": "163a31cf837d8a76b757bcfd787ed0639cca12c434e8528993bddd4c7966338f", "input_tokens": 150, "expected": "63", "text": "{\n \"answer\": \"63\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 5495, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-3-0", "object": "text_completion", "created": 1792789375, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"63\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 5495, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 150, "completion_tokens": 10, "total_tokens": 160}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 3.822009883937426} +{"case": "passkey-3-0", "route": "rngd-atom", "input_sha256": "99e6bcbba020691ee22c29fbdc8eb6c89359c3e8647f833586a2805ba089cc9a", "input_tokens": 168, "expected": "SBYZEGE7", "text": "{\n \"answer\": \"SBYZEGE7\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 17094, 41309, 9560, 36, 22, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-passkey-3-0", "object": "text_completion", "created": 1792789379, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"SBYZEGE7\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 17094, 41309, 9560, 36, 22, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 168, "completion_tokens": 14, "total_tokens": 182}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 4.104183132993057} +{"case": "korean_table-3-0", "route": "rngd-atom", "input_sha256": "80c3a0a96f8fc932afa2501e8fc3bd7d7c0a57b3d22c68a0fd8b20c7a1434527", "input_tokens": 182, "expected": "속초", "text": "{\"answer\": \"속초\"}", "tokens": [5018, 9399, 794, 330, 102130, 103719, 9388, 128009], "finish_reason": "stop", "response": {"id": "quality263-korean_table-3-0", "object": "text_completion", "created": 1792789383, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\"answer\": \"속초\"}", "token_ids": [5018, 9399, 794, 330, 102130, 103719, 9388, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 182, "completion_tokens": 8, "total_tokens": 190}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 3.7292327489703894} +{"case": "arithmetic-0-2048", "route": "rngd-atom", "input_sha256": "d5e3fc2d079f487aa54bf12711c278e030ae54da55f6cb57570ce489730d1260", "input_tokens": 2040, "expected": "96", "text": "{\n \"answer\": \"96\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 4161, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-0-2048", "object": "text_completion", "created": 1792789387, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"96\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 4161, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2040, "completion_tokens": 10, "total_tokens": 2050}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 4.325692769954912} +{"case": "passkey-0-2048", "route": "rngd-atom", "input_sha256": "6cb719c60a8d55b8ad05fa80978bbf003d37f93c9564b7425493cd5c951facfc", "input_tokens": 2045, "expected": "34HYL6W8", "text": "{\"answer\":\"34HYL6W8\"}", "tokens": [5018, 9399, 3332, 1958, 49952, 43, 21, 54, 23, 9388, 128009], "finish_reason": "stop", "response": {"id": "quality263-passkey-0-2048", "object": "text_completion", "created": 1792789391, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\"answer\":\"34HYL6W8\"}", "token_ids": [5018, 9399, 3332, 1958, 49952, 43, 21, 54, 23, 9388, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2045, "completion_tokens": 11, "total_tokens": 2056}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 4.234809871995822} +{"case": "korean_table-0-2048", "route": "rngd-atom", "input_sha256": "2de6ba8e4aab29c2b729c718bbee240e5b5b5f704d658667fb633ccff3950cdc", "input_tokens": 2044, "expected": "춘천", "text": "{\n \"answer\": \"춘천\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 122432, 101584, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-korean_table-0-2048", "object": "text_completion", "created": 1792789396, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"춘천\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 122432, 101584, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2044, "completion_tokens": 11, "total_tokens": 2055}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 4.224091391079128} +{"case": "arithmetic-1-2048", "route": "rngd-atom", "input_sha256": "ad0b2916c8ea5d6bbba7adeb75c78be114d5961607cf11fd4876214fe76f2cb8", "input_tokens": 2040, "expected": "83", "text": "{\n \"answer\": \"83\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 6069, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-1-2048", "object": "text_completion", "created": 1792789400, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"83\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 6069, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2040, "completion_tokens": 10, "total_tokens": 2050}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 4.185948702972382} +{"case": "passkey-1-2048", "route": "rngd-atom", "input_sha256": "c6f1f8c084a0f7adf16c51f2b1914dfb4da2d68c8c3a245849e38b9837183118", "input_tokens": 2043, "expected": "CNBKEFR7", "text": "{\n \"answer\": \"CNBKEFR7\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 29768, 33, 3472, 10725, 22, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-passkey-1-2048", "object": "text_completion", "created": 1792789404, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"CNBKEFR7\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 29768, 33, 3472, 10725, 22, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2043, "completion_tokens": 14, "total_tokens": 2057}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 4.5279504070058465} +{"case": "korean_table-1-2048", "route": "rngd-atom", "input_sha256": "a60cd701a8ef4007962836a35beef128c02ee2985b13b7fcbe1d21e7849ea514", "input_tokens": 2044, "expected": "목포", "text": "{\n \"answer\": \"목포\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 88708, 101796, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-korean_table-1-2048", "object": "text_completion", "created": 1792789409, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"목포\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 88708, 101796, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2044, "completion_tokens": 11, "total_tokens": 2055}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 4.401013721013442} +{"case": "arithmetic-2-2048", "route": "rngd-atom", "input_sha256": "5f0f4897424086c68ec9f858d7c2179b0335fded13c090f716857d5f9ba0bc41", "input_tokens": 2040, "expected": "43", "text": "{\n \"answer\": \"43\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 3391, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-2-2048", "object": "text_completion", "created": 1792789413, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"43\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 3391, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2040, "completion_tokens": 10, "total_tokens": 2050}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 4.31486856401898} +{"case": "passkey-2-2048", "route": "rngd-atom", "input_sha256": "793cc606a21f2d953fec07a91e099d20f3624409040251105573dab8236dad2e", "input_tokens": 2043, "expected": "WMLLRJUL", "text": "{\n \"answer\": \"WMLLRJUL\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 54, 2735, 20721, 41, 1112, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-passkey-2-2048", "object": "text_completion", "created": 1792789417, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"WMLLRJUL\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 54, 2735, 20721, 41, 1112, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2043, "completion_tokens": 14, "total_tokens": 2057}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 4.435319660929963} +{"case": "korean_table-2-2048", "route": "rngd-atom", "input_sha256": "adec171bcbf4600c8d59467a432af029bc475ce9575c1f4817eece553e37a027", "input_tokens": 2044, "expected": "전주", "text": "{\n \"answer\": \"전주\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 66965, 55430, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-korean_table-2-2048", "object": "text_completion", "created": 1792789422, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"전주\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 66965, 55430, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2044, "completion_tokens": 11, "total_tokens": 2055}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 4.244558032951318} +{"case": "arithmetic-3-2048", "route": "rngd-atom", "input_sha256": "679e19177db7acb1a7dd097a45e30f3e005511311577f7e9e9add8b11c6e9933", "input_tokens": 2040, "expected": "63", "text": "{\n \"answer\": \"63\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 5495, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-3-2048", "object": "text_completion", "created": 1792789426, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"63\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 5495, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2040, "completion_tokens": 10, "total_tokens": 2050}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 4.116499677998945} +{"case": "passkey-3-2048", "route": "rngd-atom", "input_sha256": "f4347eded9d6462727022c7ab97a9f961d43f4c44828ff7cf901fcd2a7e05510", "input_tokens": 2044, "expected": "SBYZEGE7", "text": "{\n \"answer\": \"SBYZEGE7\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 17094, 41309, 9560, 36, 22, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-passkey-3-2048", "object": "text_completion", "created": 1792789430, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"SBYZEGE7\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 17094, 41309, 9560, 36, 22, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2044, "completion_tokens": 14, "total_tokens": 2058}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 4.424890307942405} +{"case": "korean_table-3-2048", "route": "rngd-atom", "input_sha256": "6d84fc95ed8b4b55fda6af31a18bb7c32b6f19ab02d745fddd56f5ae5d74efb4", "input_tokens": 2044, "expected": "속초", "text": "{\n \"answer\": \"속초\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 102130, 103719, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-korean_table-3-2048", "object": "text_completion", "created": 1792789434, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"속초\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 102130, 103719, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2044, "completion_tokens": 11, "total_tokens": 2055}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 4.243585973978043} +{"case": "arithmetic-0-8192", "route": "rngd-atom", "input_sha256": "b5ac979ed58c0020fb09bed6e865da33798778a4e3f94b9bbd5608832c418e7b", "input_tokens": 8186, "expected": "96", "text": "{\n \"answer\": \"96\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 4161, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-0-8192", "object": "text_completion", "created": 1792789440, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"96\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 4161, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8186, "completion_tokens": 10, "total_tokens": 8196}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 5.598318464006297} +{"case": "passkey-0-8192", "route": "rngd-atom", "input_sha256": "d5cd6c30057ebdf9e65495c4a7bd757f811bf4e64860188c7357a67414f94222", "input_tokens": 8191, "expected": "34HYL6W8", "text": "34HYL6W8", "tokens": [1958, 49952, 43, 21, 54, 23, 128009], "finish_reason": "stop", "response": {"id": "quality263-passkey-0-8192", "object": "text_completion", "created": 1792789445, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "34HYL6W8", "token_ids": [1958, 49952, 43, 21, 54, 23, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8191, "completion_tokens": 7, "total_tokens": 8198}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 5.043443760019727} +{"case": "korean_table-0-8192", "route": "rngd-atom", "input_sha256": "cb0f1c7ae26517f1e9d859e6d0d24aed6e4dc2cd4d3db9fb4983c5a01d71f745", "input_tokens": 8190, "expected": "춘천", "text": "{\n \"answer\": \"춘천\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 122432, 101584, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-korean_table-0-8192", "object": "text_completion", "created": 1792789450, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"춘천\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 122432, 101584, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8190, "completion_tokens": 11, "total_tokens": 8201}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 5.323985207010992} +{"case": "arithmetic-1-8192", "route": "rngd-atom", "input_sha256": "41e87fb89c71cd493cec4801e39fccc4f625e205a485f48b1310880c5e72173f", "input_tokens": 8186, "expected": "83", "text": "{\n \"answer\": \"83\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 6069, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-1-8192", "object": "text_completion", "created": 1792789456, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"83\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 6069, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8186, "completion_tokens": 10, "total_tokens": 8196}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 5.270916817011312} +{"case": "passkey-1-8192", "route": "rngd-atom", "input_sha256": "6aec4ccde3c8e068a525f3141930fd0dd89b0066fb57ae44bf1f3695d03a9d38", "input_tokens": 8189, "expected": "CNBKEFR7", "text": "{\n \"answer\": \"CNBKEFR7\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 29768, 33, 3472, 10725, 22, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-passkey-1-8192", "object": "text_completion", "created": 1792789461, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"CNBKEFR7\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 29768, 33, 3472, 10725, 22, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8189, "completion_tokens": 14, "total_tokens": 8203}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 5.560458246036433} +{"case": "korean_table-1-8192", "route": "rngd-atom", "input_sha256": "a45e928bc070aa02a52f8f8c09fe63eb2c240804cb082b8d38395f45f19e2ac7", "input_tokens": 8190, "expected": "목포", "text": "{\n \"answer\": \"목포\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 88708, 101796, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-korean_table-1-8192", "object": "text_completion", "created": 1792789467, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"목포\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 88708, 101796, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8190, "completion_tokens": 11, "total_tokens": 8201}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 5.355753427022137} +{"case": "arithmetic-2-8192", "route": "rngd-atom", "input_sha256": "3b2113bbe36b0528ec715ac215ad332254fdcc741b55bba2fad14beb172ce03c", "input_tokens": 8186, "expected": "43", "text": "{\n \"answer\": \"43\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 3391, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-2-8192", "object": "text_completion", "created": 1792789472, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"43\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 3391, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8186, "completion_tokens": 10, "total_tokens": 8196}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 5.272638471913524} +{"case": "passkey-2-8192", "route": "rngd-atom", "input_sha256": "e8784fdb10cbfc3bc05540d0a7c9cad53c915638b3887a205ea7df911b53db53", "input_tokens": 8189, "expected": "WMLLRJUL", "text": "WMLLRJUL", "tokens": [54, 2735, 20721, 41, 1112, 128009], "finish_reason": "stop", "response": {"id": "quality263-passkey-2-8192", "object": "text_completion", "created": 1792789477, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "WMLLRJUL", "token_ids": [54, 2735, 20721, 41, 1112, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8189, "completion_tokens": 6, "total_tokens": 8195}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 4.984380865003914} +{"case": "korean_table-2-8192", "route": "rngd-atom", "input_sha256": "e2a1f3ecbfbde65119093f3e14d982ffe15fc04ff0f46e6dfb0a73aafb6bc301", "input_tokens": 8190, "expected": "전주", "text": "{\n \"answer\": \"전주\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 66965, 55430, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-korean_table-2-8192", "object": "text_completion", "created": 1792789482, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"전주\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 66965, 55430, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8190, "completion_tokens": 11, "total_tokens": 8201}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 5.326736465911381} +{"case": "arithmetic-3-8192", "route": "rngd-atom", "input_sha256": "9649667fd3e0e3f487ddf07287c5f43d01c9d0a4545f6fb163c215464c0ba727", "input_tokens": 8186, "expected": "63", "text": "{\n \"answer\": \"63\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 5495, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-3-8192", "object": "text_completion", "created": 1792789487, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"63\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 5495, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8186, "completion_tokens": 10, "total_tokens": 8196}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 5.237517460016534} +{"case": "passkey-3-8192", "route": "rngd-atom", "input_sha256": "e2301ac0d434cc1b0872c0ae4240fa21c7a89cda9a5727ff7adfbb66a6757fb3", "input_tokens": 8190, "expected": "SBYZEGE7", "text": "{\n \"answer\": \"SBYZEGE7\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 17094, 41309, 9560, 36, 22, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-passkey-3-8192", "object": "text_completion", "created": 1792789493, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"SBYZEGE7\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 17094, 41309, 9560, 36, 22, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8190, "completion_tokens": 14, "total_tokens": 8204}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 5.509376820991747} +{"case": "korean_table-3-8192", "route": "rngd-atom", "input_sha256": "ddf60d0902dc767a5b12fa8c67ed1c79a410df9d76a63d7fcccd2f21b5807ce7", "input_tokens": 8190, "expected": "속초", "text": "{\n \"answer\": \"속초\"\n}", "tokens": [517, 220, 330, 9399, 794, 330, 102130, 103719, 702, 92, 128009], "finish_reason": "stop", "response": {"id": "quality263-korean_table-3-8192", "object": "text_completion", "created": 1792789498, "model": "/tmp/serving-matrix-models/KONI-8B", "choices": [{"index": 0, "text": "{\n \"answer\": \"속초\"\n}", "token_ids": [517, 220, 330, 9399, 794, 330, 102130, 103719, 702, 92, 128009], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8190, "completion_tokens": 11, "total_tokens": 8201}}, "answer_correct": true, "format_correct": true, "elapsed_seconds": 5.303790251957253} diff --git a/opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-rngd-atom.log b/opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-rngd-atom.log new file mode 100644 index 00000000..ca71c994 --- /dev/null +++ b/opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-rngd-atom.log @@ -0,0 +1,44 @@ +ATOM_BINDING {"visible_count": 1, "physical": 1, "opened": ["/dev/rbln1"]} +{"compile_start": "rms", "width": 8192} +{"compile_done": "rms", "seconds": 0.9926180159673095} +{"compile_start": "attention", "width": 8192} +{"compile_done": "attention", "seconds": 159.76363624306396} +{"compile_start": "ffn", "width": 8192} +{"compile_done": "ffn", "seconds": 181.0874745339388} +[transformers] Ignoring clean_up_tokenization_spaces=True for BPE tokenizer TokenizersBackend. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway. +arithmetic-0-0 True True +passkey-0-0 True True +korean_table-0-0 False True +arithmetic-1-0 False True +passkey-1-0 True True +korean_table-1-0 True True +arithmetic-2-0 False True +passkey-2-0 True True +korean_table-2-0 True True +arithmetic-3-0 True True +passkey-3-0 True True +korean_table-3-0 True True +arithmetic-0-2048 True True +passkey-0-2048 True True +korean_table-0-2048 True True +arithmetic-1-2048 True True +passkey-1-2048 True True +korean_table-1-2048 True True +arithmetic-2-2048 True True +passkey-2-2048 True True +korean_table-2-2048 True True +arithmetic-3-2048 True True +passkey-3-2048 True True +korean_table-3-2048 True True +arithmetic-0-8192 True True +passkey-0-8192 False False +korean_table-0-8192 True True +arithmetic-1-8192 True True +passkey-1-8192 True True +korean_table-1-8192 True True +arithmetic-2-8192 True True +passkey-2-8192 False False +korean_table-2-8192 True True +arithmetic-3-8192 True True +passkey-3-8192 True True +korean_table-3-8192 True True diff --git a/opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-rngd-atom.producer.jsonl b/opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-rngd-atom.producer.jsonl new file mode 100644 index 00000000..373ee02f --- /dev/null +++ b/opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-rngd-atom.producer.jsonl @@ -0,0 +1,36 @@ +{"wire_id": 0, "start_s": 764310.205663944, "end_s": 764313.747463683} +{"wire_id": 1, "start_s": 764314.444059588, "end_s": 764317.61887026} +{"wire_id": 2, "start_s": 764318.615410894, "end_s": 764321.804238099} +{"wire_id": 3, "start_s": 764322.332766909, "end_s": 764325.450451806} +{"wire_id": 4, "start_s": 764326.511058926, "end_s": 764329.680428394} +{"wire_id": 5, "start_s": 764330.584889675, "end_s": 764333.768908231} +{"wire_id": 6, "start_s": 764334.270440922, "end_s": 764337.408094597} +{"wire_id": 7, "start_s": 764338.733662438, "end_s": 764341.919206591} +{"wire_id": 8, "start_s": 764342.846983858, "end_s": 764345.973689387} +{"wire_id": 9, "start_s": 764346.499901177, "end_s": 764349.662195245} +{"wire_id": 10, "start_s": 764350.322854439, "end_s": 764353.498111828} +{"wire_id": 11, "start_s": 764354.427422707, "end_s": 764357.625305937} +{"wire_id": 12, "start_s": 764358.158999047, "end_s": 764361.581704283} +{"wire_id": 13, "start_s": 764362.484181853, "end_s": 764365.750369662} +{"wire_id": 14, "start_s": 764366.719848973, "end_s": 764369.974265045} +{"wire_id": 15, "start_s": 764370.944734142, "end_s": 764374.231417856} +{"wire_id": 16, "start_s": 764375.130496973, "end_s": 764378.486376196} +{"wire_id": 17, "start_s": 764379.658906154, "end_s": 764383.091327836} +{"wire_id": 18, "start_s": 764384.060584992, "end_s": 764387.478802384} +{"wire_id": 19, "start_s": 764388.375396206, "end_s": 764391.641799639} +{"wire_id": 20, "start_s": 764392.810847048, "end_s": 764396.089482477} +{"wire_id": 21, "start_s": 764397.056828454, "end_s": 764400.268815519} +{"wire_id": 22, "start_s": 764401.173844818, "end_s": 764404.420886711} +{"wire_id": 23, "start_s": 764405.598905667, "end_s": 764408.87023542} +{"wire_id": 24, "start_s": 764409.848354097, "end_s": 764413.754574876} +{"wire_id": 25, "start_s": 764415.448555869, "end_s": 764419.020236443} +{"wire_id": 26, "start_s": 764420.490808967, "end_s": 764424.064524717} +{"wire_id": 27, "start_s": 764425.815997045, "end_s": 764429.398450834} +{"wire_id": 28, "start_s": 764431.087016393, "end_s": 764434.672567148} +{"wire_id": 29, "start_s": 764436.649793321, "end_s": 764440.244591501} +{"wire_id": 30, "start_s": 764442.00680766, "end_s": 764445.597605109} +{"wire_id": 31, "start_s": 764447.281132248, "end_s": 764450.861331982} +{"wire_id": 32, "start_s": 764452.265159569, "end_s": 764455.841403139} +{"wire_id": 33, "start_s": 764457.593142367, "end_s": 764461.149058051} +{"wire_id": 34, "start_s": 764462.833433173, "end_s": 764466.38085987} +{"wire_id": 35, "start_s": 764468.342847494, "end_s": 764471.888689302} diff --git a/opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-rngd-atom.run.json b/opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-rngd-atom.run.json new file mode 100644 index 00000000..07ae50a7 --- /dev/null +++ b/opt_prime/examples/pd/http_matrix/evidence263/quality263-koni-rngd-atom.run.json @@ -0,0 +1,16 @@ +{ + "stage": "complete", + "route": "rngd-atom", + "model": "KISTI-KONI/KONI-Llama3.1-8B-Instruct-20241024", + "runtime_commit": "3e16c33", + "torch": "2.11.0+cpu", + "tokenizer_inputs_equal": true, + "cases": 36, + "context": 9216, + "batch_size": 1, + "kv_blocks": 1, + "cache": false, + "producer_width": 8192, + "answer_correct": 31, + "format_correct": 34 +} diff --git a/opt_prime/examples/pd/http_matrix/evidence263/quality263-summary.json b/opt_prime/examples/pd/http_matrix/evidence263/quality263-summary.json index d7020eb7..c9d59007 100644 --- a/opt_prime/examples/pd/http_matrix/evidence263/quality263-summary.json +++ b/opt_prime/examples/pd/http_matrix/evidence263/quality263-summary.json @@ -212,6 +212,76 @@ } } }, + "rngd-gpu→native-atom": { + "cases": 36, + "correct_to_wrong": 1, + "wrong_to_correct": 0, + "token_equal": 35, + "changes": [ + { + "case": "passkey-3-2048", + "baseline_correct": true, + "candidate_correct": false, + "baseline_text": "{\n \"answer\": \"SBYZEGE7\"\n}", + "candidate_text": "{\n \"answer\": \"SYBZEGE7\"\n}" + } + ], + "by_context": { + "0": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + }, + "2048": { + "cases": 12, + "correct_to_wrong": 1, + "wrong_to_correct": 0, + "token_equal": 11 + }, + "8192": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + } + } + }, + "rngd-gpu→rngd-atom": { + "cases": 36, + "correct_to_wrong": 1, + "wrong_to_correct": 0, + "token_equal": 35, + "changes": [ + { + "case": "passkey-3-2048", + "baseline_correct": true, + "candidate_correct": false, + "baseline_text": "{\n \"answer\": \"SBYZEGE7\"\n}", + "candidate_text": "{\n \"answer\": \"SYBZEGE7\"\n}" + } + ], + "by_context": { + "0": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + }, + "2048": { + "cases": 12, + "correct_to_wrong": 1, + "wrong_to_correct": 0, + "token_equal": 11 + }, + "8192": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + } + } + }, "native-atom→rngd-atom": { "cases": 36, "correct_to_wrong": 0, @@ -358,6 +428,60 @@ } } }, + "rngd-gpu→native-atom": { + "cases": 36, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 36, + "changes": [], + "by_context": { + "0": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + }, + "2048": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + }, + "8192": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + } + } + }, + "rngd-gpu→rngd-atom": { + "cases": 36, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 36, + "changes": [], + "by_context": { + "0": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + }, + "2048": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + }, + "8192": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + } + } + }, "native-atom→rngd-atom": { "cases": 36, "correct_to_wrong": 0, @@ -371,140 +495,550 @@ "wrong_to_correct": 0, "token_equal": 12 }, - "2048": { - "cases": 12, - "correct_to_wrong": 0, - "wrong_to_correct": 0, - "token_equal": 12 + "2048": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + }, + "8192": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + } + } + } + } + }, + "koni": { + "routes": { + "native-gpu": { + "stage": "complete", + "cases": 36, + "answer_correct": 31, + "format_correct": 34, + "length_limited": 0, + "wrong_cases": [ + { + "case": "korean_table-0-0", + "expected": "춘천", + "text": "{\"answer\": \"부산\"}" + }, + { + "case": "arithmetic-1-0", + "expected": "83", + "text": "{\n \"answer\": \"48 + 35 = 83\"\n}" + }, + { + "case": "arithmetic-2-0", + "expected": "43", + "text": "{\n \"answer\": \"The sum of 32 and 11 is 43.\"\n}" + }, + { + "case": "passkey-0-8192", + "expected": "34HYL6W8", + "text": "34HYL6W8" + }, + { + "case": "passkey-2-8192", + "expected": "WMLLRJUL", + "text": "WMLLRJUL" + } + ] + }, + "rngd-gpu": { + "stage": "complete", + "cases": 36, + "answer_correct": 31, + "format_correct": 34, + "length_limited": 0, + "wrong_cases": [ + { + "case": "korean_table-0-0", + "expected": "춘천", + "text": "{\"answer\": \"부산\"}" + }, + { + "case": "arithmetic-1-0", + "expected": "83", + "text": "{\n \"answer\": \"48 + 35 = 83\"\n}" + }, + { + "case": "arithmetic-2-0", + "expected": "43", + "text": "{\n \"answer\": \"The sum of 32 and 11 is 43.\"\n}" + }, + { + "case": "passkey-0-8192", + "expected": "34HYL6W8", + "text": "34HYL6W8" + }, + { + "case": "passkey-2-8192", + "expected": "WMLLRJUL", + "text": "WMLLRJUL" + } + ] + }, + "native-atom": { + "stage": "complete", + "cases": 36, + "answer_correct": 31, + "format_correct": 34, + "length_limited": 0, + "wrong_cases": [ + { + "case": "korean_table-0-0", + "expected": "춘천", + "text": "{\"answer\": \"부산\"}" + }, + { + "case": "arithmetic-1-0", + "expected": "83", + "text": "{\n \"answer\": \"48 + 35 = 83\"\n}" + }, + { + "case": "arithmetic-2-0", + "expected": "43", + "text": "{\n \"answer\": \"The sum of 32 and 11 is 43.\"\n}" + }, + { + "case": "passkey-0-8192", + "expected": "34HYL6W8", + "text": "34HYL6W8" + }, + { + "case": "passkey-2-8192", + "expected": "WMLLRJUL", + "text": "WMLLRJUL" + } + ] + }, + "rngd-atom": { + "stage": "complete", + "cases": 36, + "answer_correct": 31, + "format_correct": 34, + "length_limited": 0, + "wrong_cases": [ + { + "case": "korean_table-0-0", + "expected": "춘천", + "text": "{\"answer\": \"부산\"}" + }, + { + "case": "arithmetic-1-0", + "expected": "83", + "text": "{\n \"answer\": \"48 + 35 = 83\"\n}" + }, + { + "case": "arithmetic-2-0", + "expected": "43", + "text": "{\n \"answer\": \"The sum of 32 and 11 is 43.\"\n}" + }, + { + "case": "passkey-0-8192", + "expected": "34HYL6W8", + "text": "34HYL6W8" + }, + { + "case": "passkey-2-8192", + "expected": "WMLLRJUL", + "text": "WMLLRJUL" + } + ] + } + }, + "pairs": { + "native-gpu→rngd-gpu": { + "cases": 36, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 35, + "changes": [ + { + "case": "passkey-0-2048", + "baseline_correct": true, + "candidate_correct": true, + "baseline_text": "{\n \"answer\": \"34HYL6W8\"\n}", + "candidate_text": "{\"answer\":\"34HYL6W8\"}" + } + ], + "by_context": { + "0": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + }, + "2048": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 11 + }, + "8192": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + } + } + }, + "native-gpu→native-atom": { + "cases": 36, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 36, + "changes": [], + "by_context": { + "0": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + }, + "2048": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + }, + "8192": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + } + } + }, + "native-gpu→rngd-atom": { + "cases": 36, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 35, + "changes": [ + { + "case": "passkey-0-2048", + "baseline_correct": true, + "candidate_correct": true, + "baseline_text": "{\n \"answer\": \"34HYL6W8\"\n}", + "candidate_text": "{\"answer\":\"34HYL6W8\"}" + } + ], + "by_context": { + "0": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + }, + "2048": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 11 + }, + "8192": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + } + } + }, + "rngd-gpu→native-atom": { + "cases": 36, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 35, + "changes": [ + { + "case": "passkey-0-2048", + "baseline_correct": true, + "candidate_correct": true, + "baseline_text": "{\"answer\":\"34HYL6W8\"}", + "candidate_text": "{\n \"answer\": \"34HYL6W8\"\n}" + } + ], + "by_context": { + "0": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + }, + "2048": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 11 + }, + "8192": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + } + } + }, + "rngd-gpu→rngd-atom": { + "cases": 36, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 36, + "changes": [], + "by_context": { + "0": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + }, + "2048": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + }, + "8192": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + } + } + }, + "native-atom→rngd-atom": { + "cases": 36, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 35, + "changes": [ + { + "case": "passkey-0-2048", + "baseline_correct": true, + "candidate_correct": true, + "baseline_text": "{\n \"answer\": \"34HYL6W8\"\n}", + "candidate_text": "{\"answer\":\"34HYL6W8\"}" + } + ], + "by_context": { + "0": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + }, + "2048": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 11 + }, + "8192": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + } + } + } + } + }, + "gemma": { + "routes": { + "native-gpu": { + "stage": "complete", + "cases": 36, + "answer_correct": 0, + "format_correct": 0, + "length_limited": 0, + "wrong_cases": [ + { + "case": "arithmetic-0-0", + "expected": "96", + "text": "```json\n{\n \"answer\": 96\n}\n```" + }, + { + "case": "passkey-0-0", + "expected": "34HYL6W8", + "text": "```json\n{\n \"answer\": \"34HYL6W8\"\n}\n```" + }, + { + "case": "korean_table-0-0", + "expected": "춘천", + "text": "```json\n{\n \"answer\": \"춘천\"\n}\n```" + }, + { + "case": "arithmetic-1-0", + "expected": "83", + "text": "```json\n{\n \"answer\": 83\n}\n```" + }, + { + "case": "passkey-1-0", + "expected": "CNBKEFR7", + "text": "```json\n{\n \"answer\": \"CNBKEFR7\"\n}\n```" + }, + { + "case": "korean_table-1-0", + "expected": "목포", + "text": "```json\n{\n \"answer\": \"목포\"\n}\n```" + }, + { + "case": "arithmetic-2-0", + "expected": "43", + "text": "```json\n{\n \"answer\": 33\n}\n```\n" + }, + { + "case": "passkey-2-0", + "expected": "WMLLRJUL", + "text": "```json\n{\n \"answer\": \"WMLLRJUL\"\n}\n```" + }, + { + "case": "korean_table-2-0", + "expected": "전주", + "text": "```json\n{\n \"answer\": \"전주\"\n}\n```" + }, + { + "case": "arithmetic-3-0", + "expected": "63", + "text": "```json\n{\n \"answer\": 63\n}\n```" + }, + { + "case": "passkey-3-0", + "expected": "SBYZEGE7", + "text": "```json\n{\n \"answer\": \"SBYZEGE7\"\n}\n```" + }, + { + "case": "korean_table-3-0", + "expected": "속초", + "text": "```json\n{\n \"answer\": \"부산\"\n}\n```" + }, + { + "case": "arithmetic-0-2048", + "expected": "96", + "text": "```json\n{\n \"answer\": 96\n}\n```" + }, + { + "case": "passkey-0-2048", + "expected": "34HYL6W8", + "text": "```json\n{\n \"answer\": \"34HYL6W8\"\n}\n```" + }, + { + "case": "korean_table-0-2048", + "expected": "춘천", + "text": "```json\nnull\n```" + }, + { + "case": "arithmetic-1-2048", + "expected": "83", + "text": "```json\n{\n \"answer\": 83\n}\n```" + }, + { + "case": "passkey-1-2048", + "expected": "CNBKEFR7", + "text": "```json\n{\n \"answer\": \"CNBKEFR7\"\n}\n```" + }, + { + "case": "korean_table-1-2048", + "expected": "목포", + "text": "```json\n{\n \"answer\": \"목포\"\n}\n```" + }, + { + "case": "arithmetic-2-2048", + "expected": "43", + "text": "```json\n{\n \"answer\": 33\n}\n```" + }, + { + "case": "passkey-2-2048", + "expected": "WMLLRJUL", + "text": "```json\n{\n \"answer\": \"WMLLRJUL\"\n}\n```\n" + }, + { + "case": "korean_table-2-2048", + "expected": "전주", + "text": "```json\n{\n \"answer\": \"전주\"\n}\n```" + }, + { + "case": "arithmetic-3-2048", + "expected": "63", + "text": "```json\n{\n \"answer\": 63\n}\n```" }, - "8192": { - "cases": 12, - "correct_to_wrong": 0, - "wrong_to_correct": 0, - "token_equal": 12 - } - } - } - } - }, - "koni": { - "routes": { - "native-gpu": { - "stage": "complete", - "cases": 36, - "answer_correct": 31, - "format_correct": 34, - "length_limited": 0, - "wrong_cases": [ { - "case": "korean_table-0-0", - "expected": "춘천", - "text": "{\"answer\": \"부산\"}" + "case": "passkey-3-2048", + "expected": "SBYZEGE7", + "text": "```json\n{\n \"answer\": \"SBYZEGE7\"\n}\n```" }, { - "case": "arithmetic-1-0", - "expected": "83", - "text": "{\n \"answer\": \"48 + 35 = 83\"\n}" + "case": "korean_table-3-2048", + "expected": "속초", + "text": "```json\n{\n \"answer\": \"속초\"\n}\n```" }, { - "case": "arithmetic-2-0", - "expected": "43", - "text": "{\n \"answer\": \"The sum of 32 and 11 is 43.\"\n}" + "case": "arithmetic-0-8192", + "expected": "96", + "text": "```json\n{\n \"answer\": \"96\"\n}\n```" }, { "case": "passkey-0-8192", "expected": "34HYL6W8", - "text": "34HYL6W8" + "text": "```json\n{\n \"answer\": \"34HYL6W8\"\n}\n```" }, { - "case": "passkey-2-8192", - "expected": "WMLLRJUL", - "text": "WMLLRJUL" - } - ] - }, - "rngd-gpu": { - "stage": "complete", - "cases": 36, - "answer_correct": 31, - "format_correct": 34, - "length_limited": 0, - "wrong_cases": [ - { - "case": "korean_table-0-0", + "case": "korean_table-0-8192", "expected": "춘천", - "text": "{\"answer\": \"부산\"}" + "text": "```json\n{\n \"answer\": \"부산\"\n}\n```" }, { - "case": "arithmetic-1-0", + "case": "arithmetic-1-8192", "expected": "83", - "text": "{\n \"answer\": \"48 + 35 = 83\"\n}" + "text": "```json\n{\n \"answer\": \"83\"\n}\n```" }, { - "case": "arithmetic-2-0", - "expected": "43", - "text": "{\n \"answer\": \"The sum of 32 and 11 is 43.\"\n}" + "case": "passkey-1-8192", + "expected": "CNBKEFR7", + "text": "```json\n{\n \"answer\": \"CNBKEFR7\"\n}\n```\n" }, { - "case": "passkey-0-8192", - "expected": "34HYL6W8", - "text": "34HYL6W8" + "case": "korean_table-1-8192", + "expected": "목포", + "text": "```json\n{\n \"answer\": \"목포\"\n}\n```" + }, + { + "case": "arithmetic-2-8192", + "expected": "43", + "text": "```json\n{\"answer\": \"33\"}\n```" }, { "case": "passkey-2-8192", "expected": "WMLLRJUL", - "text": "WMLLRJUL" - } - ] - }, - "native-atom": { - "stage": "not_collected" - }, - "rngd-atom": { - "stage": "not_collected" - } - }, - "pairs": { - "native-gpu→rngd-gpu": { - "cases": 36, - "correct_to_wrong": 0, - "wrong_to_correct": 0, - "token_equal": 35, - "changes": [ + "text": "```json\n{\n \"answer\": \"WMLLRJUL\"\n}\n```" + }, { - "case": "passkey-0-2048", - "baseline_correct": true, - "candidate_correct": true, - "baseline_text": "{\n \"answer\": \"34HYL6W8\"\n}", - "candidate_text": "{\"answer\":\"34HYL6W8\"}" - } - ], - "by_context": { - "0": { - "cases": 12, - "correct_to_wrong": 0, - "wrong_to_correct": 0, - "token_equal": 12 + "case": "korean_table-2-8192", + "expected": "전주", + "text": "```json\n{\n \"answer\": \"전주\"\n}\n```" }, - "2048": { - "cases": 12, - "correct_to_wrong": 0, - "wrong_to_correct": 0, - "token_equal": 11 + { + "case": "arithmetic-3-8192", + "expected": "63", + "text": "```json\n{\"answer\": \"63\"}\n```" }, - "8192": { - "cases": 12, - "correct_to_wrong": 0, - "wrong_to_correct": 0, - "token_equal": 12 + { + "case": "passkey-3-8192", + "expected": "SBYZEGE7", + "text": "```json\n{\n \"answer\": \"SBYZEGE7\"\n}\n```" + }, + { + "case": "korean_table-3-8192", + "expected": "속초", + "text": "```json\n{\n \"answer\": \"부산\"\n}\n```" } - } - } - } - }, - "gemma": { - "routes": { - "native-gpu": { + ] + }, + "rngd-gpu": { "stage": "complete", "cases": 36, "answer_correct": 0, @@ -544,7 +1078,7 @@ { "case": "arithmetic-2-0", "expected": "43", - "text": "```json\n{\n \"answer\": 33\n}\n```\n" + "text": "```json\n{\n \"answer\": 33\n}\n```" }, { "case": "passkey-2-0", @@ -584,7 +1118,7 @@ { "case": "korean_table-0-2048", "expected": "춘천", - "text": "```json\nnull\n```" + "text": "```json\n{\n \"answer\": \"춘천\"\n}\n```" }, { "case": "arithmetic-1-2048", @@ -649,7 +1183,7 @@ { "case": "arithmetic-1-8192", "expected": "83", - "text": "```json\n{\n \"answer\": \"83\"\n}\n```" + "text": "```json\n{\"answer\": \"83\"}\n```" }, { "case": "passkey-1-8192", @@ -664,7 +1198,7 @@ { "case": "arithmetic-2-8192", "expected": "43", - "text": "```json\n{\"answer\": \"33\"}\n```" + "text": "```json\n{\n \"answer\": \"33\"\n}\n```" }, { "case": "passkey-2-8192", @@ -693,7 +1227,7 @@ } ] }, - "rngd-gpu": { + "native-atom": { "stage": "complete", "cases": 36, "answer_correct": 0, @@ -773,7 +1307,7 @@ { "case": "korean_table-0-2048", "expected": "춘천", - "text": "```json\n{\n \"answer\": \"춘천\"\n}\n```" + "text": "```json\nnull\n```" }, { "case": "arithmetic-1-2048", @@ -853,7 +1387,7 @@ { "case": "arithmetic-2-8192", "expected": "43", - "text": "```json\n{\n \"answer\": \"33\"\n}\n```" + "text": "```json\n{\"answer\":\"33\"}\n```" }, { "case": "passkey-2-8192", @@ -882,9 +1416,6 @@ } ] }, - "native-atom": { - "stage": "not_collected" - }, "rngd-atom": { "stage": "not_collected" } @@ -945,6 +1476,97 @@ "token_equal": 10 } } + }, + "native-gpu→native-atom": { + "cases": 36, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 33, + "changes": [ + { + "case": "arithmetic-2-0", + "baseline_correct": false, + "candidate_correct": false, + "baseline_text": "```json\n{\n \"answer\": 33\n}\n```\n", + "candidate_text": "```json\n{\n \"answer\": 33\n}\n```" + }, + { + "case": "arithmetic-1-8192", + "baseline_correct": false, + "candidate_correct": false, + "baseline_text": "```json\n{\n \"answer\": \"83\"\n}\n```", + "candidate_text": "```json\n{\"answer\": \"83\"}\n```" + }, + { + "case": "arithmetic-2-8192", + "baseline_correct": false, + "candidate_correct": false, + "baseline_text": "```json\n{\"answer\": \"33\"}\n```", + "candidate_text": "```json\n{\"answer\":\"33\"}\n```" + } + ], + "by_context": { + "0": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 11 + }, + "2048": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + }, + "8192": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 10 + } + } + }, + "rngd-gpu→native-atom": { + "cases": 36, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 34, + "changes": [ + { + "case": "korean_table-0-2048", + "baseline_correct": false, + "candidate_correct": false, + "baseline_text": "```json\n{\n \"answer\": \"춘천\"\n}\n```", + "candidate_text": "```json\nnull\n```" + }, + { + "case": "arithmetic-2-8192", + "baseline_correct": false, + "candidate_correct": false, + "baseline_text": "```json\n{\n \"answer\": \"33\"\n}\n```", + "candidate_text": "```json\n{\"answer\":\"33\"}\n```" + } + ], + "by_context": { + "0": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + }, + "2048": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 11 + }, + "8192": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 11 + } + } } } } diff --git a/opt_prime/examples/pd/http_matrix/evidence263/research-events.jsonl b/opt_prime/examples/pd/http_matrix/evidence263/research-events.jsonl index 34815188..612516de 100644 --- a/opt_prime/examples/pd/http_matrix/evidence263/research-events.jsonl +++ b/opt_prime/examples/pd/http_matrix/evidence263/research-events.jsonl @@ -6,3 +6,4 @@ {"id": "d4ffad67020746a0b4b57be8e175f205", "ts": "2026-10-24T05:38:32+09:00", "task": "Structured answer compatibility (aicomp #263)", "result": "All288 GPU responses complete; exacttoken pairs Q06=34/36,Q4=36/36,KONI=35/36,Gemma=32/36. Q06ATOM pair36/36 token equal; Q4nativeATOM36/36 structured score. All21 source checkpoint/config/tokenizer files SHA256-identical across hosts.", "next": "Finish five remaining ATOM routes from live owned queuePID1292321; preserve format/content distinction and strict historical token failures", "state": "active", "done": false, "blocker": "", "metrics": {}, "params": {}, "artifacts": [{"path": "analysis/serving-matrix-201/quality263-summary.json", "size": 27683, "sha256": "0cf7034ef6912788"}, {"path": "analysis/serving-matrix-201/quality263-model-identity.json", "size": 766, "sha256": "8a208a90a6cbc8c6"}], "experiments": ["experiments/2026-10-24-heldout-quality263.md"], "labels": ["experiment"], "review": false, "todos": [], "note": "", "agent": "codex", "host": "xeon-max", "branch": "main", "commit": "351d39d-dirty"} {"id": "ba772c50c9b14714bc71380ceb9980be", "ts": "2026-10-24T05:39:35+09:00", "task": "Structured answer compatibility (aicomp #263)", "result": "Published all GPU pairs and Q06ATOM pair in PR264 d8e6d0a; source21files SHA256 identical. Published audit rerun matches saved summary after correcting manifest filename. Local accelerator jobs exited; owned ATOM queue1292321 remains active.", "next": "Wait for actual Qwen4/KONI/Gemma ATOM completions, collect evidence and audit remaining compatibility requirements", "state": "active", "done": false, "blocker": "", "metrics": {}, "params": {}, "artifacts": [], "experiments": ["experiments/2026-10-24-heldout-quality263.md"], "labels": ["experiment"], "review": false, "todos": [], "note": "", "agent": "codex", "host": "xeon-max", "branch": "main", "commit": "351d39d-dirty"} {"id": "74ff20284c5f4c01b485d538923b2200", "ts": "2026-10-24T05:54:06+09:00", "task": "Structured answer compatibility (aicomp #263)", "result": "Qwen3-4B allfour routes complete:36/36 answers each, every paired token sequence exact. Remaining ATOM routes KONI/Gemma running in owned queue1292321. Existing long-free-generation issue256 remains open.", "next": "Finish KONI/Gemma ATOM; publish four-model final quality matrix; continue isolated265layer-progress integration", "state": "active", "done": false, "blocker": "", "metrics": {}, "params": {"model": "Qwen3-4B", "cases": "36"}, "artifacts": [{"path": "analysis/serving-matrix-201/quality263-summary.json", "size": 29294, "sha256": "d5f03ba245ebf8b1"}], "experiments": ["experiments/2026-10-24-heldout-quality263.md"], "labels": ["experiment"], "review": false, "todos": [], "note": "", "agent": "codex", "host": "xeon-max", "branch": "main", "commit": "351d39d-dirty"} +{"id": "3dac1bdaaf9c45ea869042501f65e19b", "ts": "2026-10-24T06:12:59+09:00", "task": "Structured answer compatibility (aicomp #263)", "result": "All six route pairs audited: Q06 passkey-3-2048 correct on RNGD-GPU but wrong on bothATOM routes (SBYZEGE7→SYBZEGE7). KONI allfour31/36 scores; source-matched decode pairs36/36 exact. Only Gemma RNGD-ATOM remains active.", "next": "Finish Gemma; preserve counterexample and distinguish source/decoder numerical causes with matched-KV controls if needed", "state": "active", "done": false, "blocker": "", "metrics": {}, "params": {}, "artifacts": [{"path": "analysis/serving-matrix-201/quality263-summary.json", "size": 48519, "sha256": "528c72c6bd0ec537"}], "experiments": ["experiments/2026-10-24-heldout-quality263.md"], "labels": ["analysis"], "review": false, "todos": [], "note": "", "agent": "codex", "host": "xeon-max", "branch": "main", "commit": "351d39d-dirty"} diff --git a/opt_prime/examples/pd/http_matrix/evidence263/summarize_quality263.py b/opt_prime/examples/pd/http_matrix/evidence263/summarize_quality263.py index a65c7bd6..a21dbcbf 100644 --- a/opt_prime/examples/pd/http_matrix/evidence263/summarize_quality263.py +++ b/opt_prime/examples/pd/http_matrix/evidence263/summarize_quality263.py @@ -1,4 +1,5 @@ """Audit complete runs against frozen fixtures; preserve pending/failed routes explicitly.""" +from itertools import combinations import hashlib import importlib.util import json @@ -42,8 +43,7 @@ 'format_correct':sum(r['format_correct'] for r in rows), 'length_limited':sum(r['finish_reason']=='length' for r in rows), 'wrong_cases':[dict(case=r['case'],expected=r['expected'],text=r['text']) for r in rows if not r['answer_correct']]} - for baseline,candidate in [('native-gpu','rngd-gpu'),('native-gpu','native-atom'), - ('native-gpu','rngd-atom'),('native-atom','rngd-atom')]: + for baseline,candidate in combinations(['native-gpu','rngd-gpu','native-atom','rngd-atom'],2): if baseline not in completed or candidate not in completed: continue a,b=completed[baseline],completed[candidate] other={r['case']:r for r in b} diff --git a/opt_prime/examples/pd/http_matrix/experiments/2026-10-24-heldout-quality263.md b/opt_prime/examples/pd/http_matrix/experiments/2026-10-24-heldout-quality263.md index c725b7f0..4ff585c7 100644 --- a/opt_prime/examples/pd/http_matrix/experiments/2026-10-24-heldout-quality263.md +++ b/opt_prime/examples/pd/http_matrix/experiments/2026-10-24-heldout-quality263.md @@ -78,3 +78,9 @@ Both hosts' complete source weight shards, config.json, generation_config.json a Auditor summarize_quality263.py verifies fixture-file hashes, case/input-hash correspondence, response fields, scores recomputed from raw text, and paired transitions. Pending routes remain explicitly not_collected/starting; they are not treated as passes. Qwen3-4B allfour routes completed: each36/36 structured answers and36/36 format, every paired token sequence identical acrossGPU/RNGD/ATOM. Remaining ATOM routes: KONI and Gemma. This does not erase the prior different long-free-generation fixture failure#256. + +## Full pairwise audit (primary prompts/scorer unchanged) + +Compare all six unordered route pairs, not only pairs anchored on nativeGPU. This exposes a Qwen0.6 counterexample: passkey-3-2048 is correct`SBYZEGE7` on RNGD→GPU, but nativeATOM and RNGD→ATOM emit`SYBZEGE7`. Thus each ATOM route has one correct→wrong transition relative to RNGD→GPU, despite no such transition relative to nativeGPU (which also answered this case incorrectly). Do not claim universal cross-route quality equivalence or infer the failing operator without matched-KV controls. + +KONI allfour routes completed:31/36 structured answers each; nativeGPU/nativeATOM36/36 exacttokens; RNGD→GPU/RNGD→ATOM36/36 exacttokens. Native vsRNGD pair35/36; the differing text only changes JSON whitespace. Gemma nativeATOM completed with code-fenced responses (strictscore0); RNGD→ATOM remains running. From c28d05622a044b0d2544bba17f9ebd1e5b8c940f Mon Sep 17 00:00:00 2001 From: gspark Date: Sat, 24 Oct 2026 06:29:49 +0900 Subject: [PATCH 6/7] test: complete four-model held-out route audit --- .../pd/http_matrix/.research/events.jsonl | 1 + opt_prime/examples/pd/http_matrix/STATUS.md | 4 + .../quality263-gemma-rngd-atom.jsonl | 36 +++ .../quality263-gemma-rngd-atom.log | 37 +++ .../quality263-gemma-rngd-atom.producer.jsonl | 36 +++ .../quality263-gemma-rngd-atom.run.json | 16 + .../quality263-pod-after-cleanup.json | 5 + .../evidence263/quality263-summary.json | 306 +++++++++++++++++- .../2026-10-24-heldout-quality263.md | 15 + 9 files changed, 455 insertions(+), 1 deletion(-) create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-rngd-atom.jsonl create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-rngd-atom.log create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-rngd-atom.producer.jsonl create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-rngd-atom.run.json create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-pod-after-cleanup.json diff --git a/opt_prime/examples/pd/http_matrix/.research/events.jsonl b/opt_prime/examples/pd/http_matrix/.research/events.jsonl index f883ccc0..98d83845 100644 --- a/opt_prime/examples/pd/http_matrix/.research/events.jsonl +++ b/opt_prime/examples/pd/http_matrix/.research/events.jsonl @@ -8,3 +8,4 @@ {"id": "c41ea1b2753146b091f85134832aa680", "ts": "2026-10-24T04:44:49+09:00", "task": "Four-model serving matrix (aicomp #201)", "result": "Prepared independent live GPU lifecycle validation branch on32f18ff with executed harness, native-call observer and completed Qwen0.6/Qwen4 raw evidence; KONI remains live. CLI help verified; production runtime unchanged.", "next": "Publish draft evidence PR and complete KONI/Gemma lifecycle runs", "state": "active", "done": false, "blocker": "", "metrics": {}, "params": {}, "artifacts": [], "experiments": ["experiments/2026-10-24-live-gpu-fault201.md"], "labels": ["benchmark"], "review": false, "todos": [], "note": "", "agent": "codex", "host": "xeon-max", "branch": "main", "commit": "351d39d-dirty"} {"id": "2919d0420ffc49ceacc9aa8982bf4859", "ts": "2026-10-24T04:48:11+09:00", "task": "Four-model serving matrix (aicomp #201)", "result": "All4 live GPU lifecycle runs completed passed=true, including KONI/Gemma: cancellation/reuse, producer death503/readiness, slab unlink and replacement exact32-token controls. Started separate512-token concurrent-decode isolation suite after prior queue terminal and device-idle checks.", "next": "Publish remaining lifecycle evidence; verify currently decoding requests survive producer failure across4models", "state": "active", "done": false, "blocker": "", "metrics": {}, "params": {}, "artifacts": [{"path": "analysis/serving-matrix-201/local201-koni-faults.json", "size": 1003, "sha256": "5ef1011fb2c2a4c6"}, {"path": "analysis/serving-matrix-201/local201-gemma-faults.json", "size": 1016, "sha256": "e707fb1f52adbb1a"}], "experiments": ["experiments/2026-10-24-live-gpu-fault201.md"], "labels": ["benchmark"], "review": false, "todos": [], "note": "", "agent": "codex", "host": "xeon-max", "branch": "main", "commit": "351d39d-dirty"} {"id": "eb53d939aca44b85ab4b6fd602754651", "ts": "2026-10-24T04:58:02+09:00", "task": "Four-model serving matrix (aicomp #201)", "result": "All4 GPU concurrent-decode isolation tests passed512exact tokens after producer killed at token2, plus cancellation/503/slab-unlink/replacement. KONI paddedGPU control completed: changes output but does NOT reproduce short RNGD output; padding-only explanation rejected.", "next": "Publish all4isolation evidence; capture short KONI KV for identical-prefix numerical analysis", "state": "active", "done": false, "blocker": "", "metrics": {}, "params": {}, "artifacts": [{"path": "analysis/serving-matrix-201/koni-short-padding-diagnostic.json", "size": 5903, "sha256": "f67039817a78849a"}], "experiments": ["experiments/2026-10-24-live-gpu-fault201.md"], "labels": ["benchmark"], "review": false, "todos": [], "note": "", "agent": "codex", "host": "xeon-max", "branch": "main", "commit": "351d39d-dirty"} +{"id": "ce47a4a232ae4f6dbb19f8738f8b2364", "ts": "2026-10-24T06:29:28+09:00", "task": "Structured answer compatibility (aicomp #263)", "result": "Completed all576 responses, normal EOS and no inference failures. Qwen3-4B allfour routes36/36 answers and token-exact; Q06 and Gemma retain cross-route content counterexamples. Final published artifact auditor reproduces summary exactly; owned Pod absent.", "next": "Review PR264; investigate content differences with matched-KV controls; complete source-side streaming in #265", "state": "active", "done": false, "blocker": "", "metrics": {}, "params": {"models": "4", "routes": "4", "cases_per_model": "36"}, "artifacts": ["evidence263/quality263-summary.json"], "experiments": ["experiments/2026-10-24-heldout-quality263.md"], "labels": ["experiment", "rngd"], "review": true, "todos": [], "note": "", "agent": "codex", "host": "xeon-max", "branch": "main", "commit": "351d39d-dirty"} diff --git a/opt_prime/examples/pd/http_matrix/STATUS.md b/opt_prime/examples/pd/http_matrix/STATUS.md index 8440687d..a8237d8e 100644 --- a/opt_prime/examples/pd/http_matrix/STATUS.md +++ b/opt_prime/examples/pd/http_matrix/STATUS.md @@ -59,3 +59,7 @@ Allfour live GPU isolation runs pass: active512-token decode survives other-requ #263 Qwen4allfour routes complete:each36/36 structured answers and all36 token sequences equal acrossGPU/RNGD/ATOM. KONI/GemmaATOM pending; historical#256 remains open. #263 six-pair audit: Q06 passkey-3-2048 correct on RNGD→GPU but wrong on bothATOM routes; no universal quality-equivalence claim. KONI allfour complete31/36 scores; nativeGPU/nativeATOM andRNGD-GPU/RNGD-ATOM each36/36 exacttokens. OnlyGemma RNGD-ATOM pending. + +## Final held-out evaluation (#263) + +All 576 responses completed normally. Qwen3-4B: 36/36 structured answers and exact tokens on all four routes. Qwen3-0.6B and Gemma retain cross-route content counterexamples; all six route pairs are audited. Final published artifact auditor reproduces the summary exactly. Owned evaluation Pod absent. See evidence263 and the heldout-quality263 experiment; numerical compatibility and source streaming remain open. diff --git a/opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-rngd-atom.jsonl b/opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-rngd-atom.jsonl new file mode 100644 index 00000000..fd15e53e --- /dev/null +++ b/opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-rngd-atom.jsonl @@ -0,0 +1,36 @@ +{"case": "arithmetic-0-0", "route": "rngd-atom", "input_sha256": "ca6bce506a59e3ff2ea456d58be66d889b684a8d3c0d8264ea1adf319eba3215", "input_tokens": 38, "expected": "96", "text": "```json\n{\n \"answer\": 96\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 236743, 236819, 236825, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-0-0", "object": "text_completion", "created": 1792790367, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": 96\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 236743, 236819, 236825, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 38, "completion_tokens": 17, "total_tokens": 55}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 1.811609800090082} +{"case": "passkey-0-0", "route": "rngd-atom", "input_sha256": "e6af0e26e172ab6739b897c7e5886bbd244a67d68e688c29251f38f0b623c516", "input_tokens": 57, "expected": "34HYL6W8", "text": "```json\n{\n \"answer\": \"34HYL6W8\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 236800, 236812, 32207, 236798, 236825, 236824, 236828, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-passkey-0-0", "object": "text_completion", "created": 1792790369, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"34HYL6W8\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 236800, 236812, 32207, 236798, 236825, 236824, 236828, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 57, "completion_tokens": 23, "total_tokens": 80}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 1.8206225900212303} +{"case": "korean_table-0-0", "route": "rngd-atom", "input_sha256": "ea4ffcca39f8322f4ad91bec1ebc6bd77d5e11cfa2c10de1e5e0491a0c771c04", "input_tokens": 68, "expected": "춘천", "text": "```json\n{\n \"answer\": \"춘천\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 242349, 239360, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-korean_table-0-0", "object": "text_completion", "created": 1792790371, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"춘천\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 242349, 239360, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 68, "completion_tokens": 18, "total_tokens": 86}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 1.7330638739513233} +{"case": "arithmetic-1-0", "route": "rngd-atom", "input_sha256": "9f6534dcba2389afa359614d713c5ccd0c632c8a3d737e948f5744236216751c", "input_tokens": 38, "expected": "83", "text": "```json\n{\n \"answer\": 83\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 236743, 236828, 236800, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-1-0", "object": "text_completion", "created": 1792790372, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": 83\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 236743, 236828, 236800, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 38, "completion_tokens": 17, "total_tokens": 55}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 1.7298622420057654} +{"case": "passkey-1-0", "route": "rngd-atom", "input_sha256": "8110d36233ff6302384e08812339b32a6505ad4f29827c8fcbc3cc5e6f7f8143", "input_tokens": 54, "expected": "CNBKEFR7", "text": "```json\n{\n \"answer\": \"CNBKEFR7\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 9353, 236799, 11998, 11869, 236832, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-passkey-1-0", "object": "text_completion", "created": 1792790374, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"CNBKEFR7\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 9353, 236799, 11998, 11869, 236832, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 54, "completion_tokens": 21, "total_tokens": 75}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 1.7933141579851508} +{"case": "korean_table-1-0", "route": "rngd-atom", "input_sha256": "e4ca0aa029d8e1569bb761ba0f1d8945c6ff89c30fc5676f2f5a70ebebebded6", "input_tokens": 68, "expected": "목포", "text": "```json\n{\n \"answer\": \"목포\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 239136, 238709, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-korean_table-1-0", "object": "text_completion", "created": 1792790376, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"목포\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 239136, 238709, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 68, "completion_tokens": 18, "total_tokens": 86}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 1.759647281956859} +{"case": "arithmetic-2-0", "route": "rngd-atom", "input_sha256": "86c0f5977a407fb0b704b381d799c8ec10e3c96c8c943eb9d09f69f90bcb7337", "input_tokens": 38, "expected": "43", "text": "```json\n{\n \"answer\": 33\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 236743, 236800, 236800, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-2-0", "object": "text_completion", "created": 1792790378, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": 33\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 236743, 236800, 236800, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 38, "completion_tokens": 17, "total_tokens": 55}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 1.7429394649807364} +{"case": "passkey-2-0", "route": "rngd-atom", "input_sha256": "beaa3fab948d71c3e9b2793b3fe179ce4cf2eeee17923a2f36cac0da494b4e0e", "input_tokens": 53, "expected": "WMLLRJUL", "text": "```json\n{\n \"answer\": \"WMLLRJUL\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 15378, 2182, 136559, 4057, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-passkey-2-0", "object": "text_completion", "created": 1792790380, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"WMLLRJUL\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 15378, 2182, 136559, 4057, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 53, "completion_tokens": 20, "total_tokens": 73}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 1.7842714600265026} +{"case": "korean_table-2-0", "route": "rngd-atom", "input_sha256": "6c40dd52b87165747a3f02db798f2ae066b85043ed5d362ce32ca37107517bc5", "input_tokens": 68, "expected": "전주", "text": "```json\n{\n \"answer\": \"전주\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 237773, 237754, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-korean_table-2-0", "object": "text_completion", "created": 1792790381, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"전주\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 237773, 237754, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 68, "completion_tokens": 18, "total_tokens": 86}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 1.7590842889621854} +{"case": "arithmetic-3-0", "route": "rngd-atom", "input_sha256": "fef18b91d4de61b1ee4714defa961ff75523d237e32fd6d1a9aa05105b64a23e", "input_tokens": 38, "expected": "63", "text": "```json\n{\n \"answer\": 63\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 236743, 236825, 236800, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-3-0", "object": "text_completion", "created": 1792790383, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": 63\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 236743, 236825, 236800, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 38, "completion_tokens": 17, "total_tokens": 55}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 1.7486669650534168} +{"case": "passkey-3-0", "route": "rngd-atom", "input_sha256": "9d26ce509899d1c3ee7d654c2a5b9eef427fe3f062445022f1c12dc1c7a5d433", "input_tokens": 54, "expected": "SBYZEGE7", "text": "```json\n{\n \"answer\": \"SBYZEGE7\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 236773, 19470, 7942, 8809, 236832, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-passkey-3-0", "object": "text_completion", "created": 1792790385, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"SBYZEGE7\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 236773, 19470, 7942, 8809, 236832, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 54, "completion_tokens": 21, "total_tokens": 75}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 1.8138587389839813} +{"case": "korean_table-3-0", "route": "rngd-atom", "input_sha256": "80e75874d5f77f73b151cd532d78d8e353da8fa375ce6aa622fb4edaa66b2c11", "input_tokens": 68, "expected": "속초", "text": "```json\n{\n \"answer\": \"부산\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 184764, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-korean_table-3-0", "object": "text_completion", "created": 1792790387, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"부산\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 184764, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 68, "completion_tokens": 17, "total_tokens": 85}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 1.7497588880360126} +{"case": "arithmetic-0-2048", "route": "rngd-atom", "input_sha256": "af4879aef2e6d64f89abce61ba827a7147ce7aa68f131b5059730f6daedfbf89", "input_tokens": 2048, "expected": "96", "text": "```json\n{\n \"answer\": 96\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 236743, 236819, 236825, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-0-2048", "object": "text_completion", "created": 1792790388, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": 96\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 236743, 236819, 236825, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2048, "completion_tokens": 17, "total_tokens": 2065}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 1.8569042610470206} +{"case": "passkey-0-2048", "route": "rngd-atom", "input_sha256": "ec3792ad17aed14871774cb4bfb238a098a5fbfab84f26321dbaffef4b4bc183", "input_tokens": 2037, "expected": "34HYL6W8", "text": "```json\n{\n \"answer\": \"34HYL6W8\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 236800, 236812, 32207, 236798, 236825, 236824, 236828, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-passkey-0-2048", "object": "text_completion", "created": 1792790390, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"34HYL6W8\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 236800, 236812, 32207, 236798, 236825, 236824, 236828, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2037, "completion_tokens": 23, "total_tokens": 2060}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 1.8641267060302198} +{"case": "korean_table-0-2048", "route": "rngd-atom", "input_sha256": "3f9e6fea06717095d1a90b92ac31397cb680a716891bcdd1d20b251fb5511eb5", "input_tokens": 2048, "expected": "춘천", "text": "```json\nnull\n```", "tokens": [2717, 3723, 107, 6455, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-korean_table-0-2048", "object": "text_completion", "created": 1792790392, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\nnull\n```", "token_ids": [2717, 3723, 107, 6455, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2048, "completion_tokens": 7, "total_tokens": 2055}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 1.700473576085642} +{"case": "arithmetic-1-2048", "route": "rngd-atom", "input_sha256": "c705989bcfc419d3aa14ab0b042b6e1ae5353866cd792b3c7891b76451e8ee5f", "input_tokens": 2048, "expected": "83", "text": "```json\n{\n \"answer\": 83\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 236743, 236828, 236800, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-1-2048", "object": "text_completion", "created": 1792790394, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": 83\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 236743, 236828, 236800, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2048, "completion_tokens": 17, "total_tokens": 2065}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 1.8080820640316233} +{"case": "passkey-1-2048", "route": "rngd-atom", "input_sha256": "e1fbe20650a58df3fd16efb5b61c1e1d2a45ea5e677d760cea1ad435789ccec6", "input_tokens": 2034, "expected": "CNBKEFR7", "text": "```json\n{\n \"answer\": \"CNBKEFR7\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 9353, 236799, 11998, 11869, 236832, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-passkey-1-2048", "object": "text_completion", "created": 1792790396, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"CNBKEFR7\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 9353, 236799, 11998, 11869, 236832, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2034, "completion_tokens": 21, "total_tokens": 2055}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 1.8839425459736958} +{"case": "korean_table-1-2048", "route": "rngd-atom", "input_sha256": "d5a80b27cd41a44c089a38a4654dc064496f7e833b906493f407cf49c12879f5", "input_tokens": 2048, "expected": "목포", "text": "```json\n{\n \"answer\": \"목포\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 239136, 238709, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-korean_table-1-2048", "object": "text_completion", "created": 1792790398, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"목포\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 239136, 238709, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2048, "completion_tokens": 18, "total_tokens": 2066}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 1.850483421003446} +{"case": "arithmetic-2-2048", "route": "rngd-atom", "input_sha256": "3718106299b7ba6fbcead43904b7e934e114eeb3f6164df3c7fe1db11599a006", "input_tokens": 2048, "expected": "43", "text": "```json\n{\n \"answer\": 33\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 236743, 236800, 236800, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-2-2048", "object": "text_completion", "created": 1792790399, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": 33\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 236743, 236800, 236800, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2048, "completion_tokens": 17, "total_tokens": 2065}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 1.8077803840860724} +{"case": "passkey-2-2048", "route": "rngd-atom", "input_sha256": "31115153f401db2cb1466d72cf519d69168b22f449778893b0c3acca34cea0fb", "input_tokens": 2048, "expected": "WMLLRJUL", "text": "```json\n{\n \"answer\": \"WMLLRJUL\"\n}\n```\n", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 15378, 2182, 136559, 4057, 236775, 107, 236783, 107, 2717, 107, 106], "finish_reason": "stop", "response": {"id": "quality263-passkey-2-2048", "object": "text_completion", "created": 1792790401, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"WMLLRJUL\"\n}\n```\n", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 15378, 2182, 136559, 4057, 236775, 107, 236783, 107, 2717, 107, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2048, "completion_tokens": 21, "total_tokens": 2069}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 1.8469931479776278} +{"case": "korean_table-2-2048", "route": "rngd-atom", "input_sha256": "9599db7b28431faf2421bf2180833e5a4aef98483a7254721f1ec83c92fd46f7", "input_tokens": 2048, "expected": "전주", "text": "```json\n{\n \"answer\": \"전주\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 237773, 237754, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-korean_table-2-2048", "object": "text_completion", "created": 1792790403, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"전주\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 237773, 237754, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2048, "completion_tokens": 18, "total_tokens": 2066}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 1.8555202840361744} +{"case": "arithmetic-3-2048", "route": "rngd-atom", "input_sha256": "251532c0d3e1841162c6c7e2dcc8bd256cc44a258af9daae4aa3d23024810d7c", "input_tokens": 2048, "expected": "63", "text": "```json\n{\n \"answer\": 63\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 236743, 236825, 236800, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-3-2048", "object": "text_completion", "created": 1792790405, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": 63\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 236743, 236825, 236800, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2048, "completion_tokens": 17, "total_tokens": 2065}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 1.8387709659291431} +{"case": "passkey-3-2048", "route": "rngd-atom", "input_sha256": "1bfc4b6311f887ae68cf363ab09f2e0115e9a5bce4dee802a84068ee11ea73ee", "input_tokens": 2034, "expected": "SBYZEGE7", "text": "```json\n{\n \"answer\": \"SBYZEGE7\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 236773, 19470, 7942, 8809, 236832, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-passkey-3-2048", "object": "text_completion", "created": 1792790407, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"SBYZEGE7\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 236773, 19470, 7942, 8809, 236832, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2034, "completion_tokens": 21, "total_tokens": 2055}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 1.8615539559395984} +{"case": "korean_table-3-2048", "route": "rngd-atom", "input_sha256": "3f1148302c3113bb35b82b8c90495e402f0fd8a6ee32cc13ab54f2e88a5bc569", "input_tokens": 2048, "expected": "속초", "text": "```json\n{\n \"answer\": \"속초\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 238701, 239378, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-korean_table-3-2048", "object": "text_completion", "created": 1792790409, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"속초\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 238701, 239378, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 2048, "completion_tokens": 18, "total_tokens": 2066}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 1.837587215937674} +{"case": "arithmetic-0-8192", "route": "rngd-atom", "input_sha256": "8f09d7d4d70782f016ec9530e14fd8992c798993340fd89dc91a1c1dd8f98686", "input_tokens": 8183, "expected": "96", "text": "```json\n{\n \"answer\": \"96\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 236819, 236825, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-0-8192", "object": "text_completion", "created": 1792790411, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"96\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 236819, 236825, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8183, "completion_tokens": 18, "total_tokens": 8201}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 2.123659125994891} +{"case": "passkey-0-8192", "route": "rngd-atom", "input_sha256": "04b383a0a4fbcb099f7b148e3e949fde72c21f6231bbec3dce68f53027894edf", "input_tokens": 8187, "expected": "34HYL6W8", "text": "```json\n{\n \"answer\": \"34HYL6W8\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 236800, 236812, 32207, 236798, 236825, 236824, 236828, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-passkey-0-8192", "object": "text_completion", "created": 1792790413, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"34HYL6W8\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 236800, 236812, 32207, 236798, 236825, 236824, 236828, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8187, "completion_tokens": 23, "total_tokens": 8210}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 2.095498924027197} +{"case": "korean_table-0-8192", "route": "rngd-atom", "input_sha256": "e770191b7695f0d43ce4a12097d1d2ef0ae30f8c39118e7104e4d21047a4d5c4", "input_tokens": 8183, "expected": "춘천", "text": "```json\n{\n \"answer\": \"부산\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 184764, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-korean_table-0-8192", "object": "text_completion", "created": 1792790415, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"부산\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 184764, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8183, "completion_tokens": 17, "total_tokens": 8200}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 2.0116345239803195} +{"case": "arithmetic-1-8192", "route": "rngd-atom", "input_sha256": "ae1469d2d3a14cf7da2fa13c1f8128325479c8db9c7bb8f6123fbecd93e3e4cf", "input_tokens": 8183, "expected": "83", "text": "```json\n{\"answer\": \"83\"}\n```", "tokens": [2717, 3723, 107, 14937, 14433, 1083, 623, 236828, 236800, 25938, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-1-8192", "object": "text_completion", "created": 1792790417, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\"answer\": \"83\"}\n```", "token_ids": [2717, 3723, 107, 14937, 14433, 1083, 623, 236828, 236800, 25938, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8183, "completion_tokens": 13, "total_tokens": 8196}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 1.9643656610278413} +{"case": "passkey-1-8192", "route": "rngd-atom", "input_sha256": "e59b6d8def078023ad5694672bb2bbec6994e459bc39c7cfab95eb2a0418008a", "input_tokens": 8184, "expected": "CNBKEFR7", "text": "```json\n{\n \"answer\": \"CNBKEFR7\"\n}\n```\n", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 9353, 236799, 11998, 11869, 236832, 236775, 107, 236783, 107, 2717, 107, 106], "finish_reason": "stop", "response": {"id": "quality263-passkey-1-8192", "object": "text_completion", "created": 1792790419, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"CNBKEFR7\"\n}\n```\n", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 9353, 236799, 11998, 11869, 236832, 236775, 107, 236783, 107, 2717, 107, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8184, "completion_tokens": 22, "total_tokens": 8206}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 2.074868489988148} +{"case": "korean_table-1-8192", "route": "rngd-atom", "input_sha256": "9192491b17b6fa3a60efa8f084df443ffe2522a806a7eb6abbf9904ad6a5a50e", "input_tokens": 8183, "expected": "목포", "text": "```json\n{\n \"answer\": \"목포\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 239136, 238709, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-korean_table-1-8192", "object": "text_completion", "created": 1792790421, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"목포\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 239136, 238709, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8183, "completion_tokens": 18, "total_tokens": 8201}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 2.0178780399728566} +{"case": "arithmetic-2-8192", "route": "rngd-atom", "input_sha256": "04e5f9a4c4bdf5b4da2accc13cc755f3c422b8ff16dc60383234ed0ce0c1d744", "input_tokens": 8183, "expected": "43", "text": "```json\n{\"answer\":\"33\"}\n```", "tokens": [2717, 3723, 107, 14937, 14433, 12375, 236800, 236800, 25938, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-2-8192", "object": "text_completion", "created": 1792790423, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\"answer\":\"33\"}\n```", "token_ids": [2717, 3723, 107, 14937, 14433, 12375, 236800, 236800, 25938, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8183, "completion_tokens": 12, "total_tokens": 8195}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 1.9604339249199256} +{"case": "passkey-2-8192", "route": "rngd-atom", "input_sha256": "bf1531f20519b1d1162a4230d9835ed8e395f12c9c94444744271828d7dea36a", "input_tokens": 8183, "expected": "WMLLRJUL", "text": "```json\n{\n \"answer\": \"WMLLRJUL\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 15378, 2182, 136559, 4057, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-passkey-2-8192", "object": "text_completion", "created": 1792790425, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"WMLLRJUL\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 15378, 2182, 136559, 4057, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8183, "completion_tokens": 20, "total_tokens": 8203}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 2.0371317849494517} +{"case": "korean_table-2-8192", "route": "rngd-atom", "input_sha256": "bc8f317a6ec4db698ed1f794e9d056bc62a289ce6f49ff08b918454cf800456e", "input_tokens": 8183, "expected": "전주", "text": "```json\n{\n \"answer\": \"전주\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 237773, 237754, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-korean_table-2-8192", "object": "text_completion", "created": 1792790427, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"전주\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 237773, 237754, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8183, "completion_tokens": 18, "total_tokens": 8201}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 2.0203045269008726} +{"case": "arithmetic-3-8192", "route": "rngd-atom", "input_sha256": "a1fb40e56504a08b4fea4da371f0fce7183ca271335fa8039921ecf9f752868c", "input_tokens": 8183, "expected": "63", "text": "```json\n{\"answer\": \"63\"}\n```", "tokens": [2717, 3723, 107, 14937, 14433, 1083, 623, 236825, 236800, 25938, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-arithmetic-3-8192", "object": "text_completion", "created": 1792790429, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\"answer\": \"63\"}\n```", "token_ids": [2717, 3723, 107, 14937, 14433, 1083, 623, 236825, 236800, 25938, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8183, "completion_tokens": 13, "total_tokens": 8196}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 1.9378423700109124} +{"case": "passkey-3-8192", "route": "rngd-atom", "input_sha256": "f3ba91ce9a3a993574e925264bc3207381e9bfb0710fbf6a2fa3f424774268b1", "input_tokens": 8184, "expected": "SBYZEGE7", "text": "```json\n{\n \"answer\": \"SBYZEGE7\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 236773, 19470, 7942, 8809, 236832, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-passkey-3-8192", "object": "text_completion", "created": 1792790431, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"SBYZEGE7\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 236773, 19470, 7942, 8809, 236832, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8184, "completion_tokens": 21, "total_tokens": 8205}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 2.0807331250980496} +{"case": "korean_table-3-8192", "route": "rngd-atom", "input_sha256": "4ef06a4874f17e50cec21d5f2611a9ff97e8c19b35a1446fa0cafd1f57c3fb40", "input_tokens": 8183, "expected": "속초", "text": "```json\n{\n \"answer\": \"부산\"\n}\n```", "tokens": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 184764, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop", "response": {"id": "quality263-korean_table-3-8192", "object": "text_completion", "created": 1792790433, "model": "/tmp/serving-matrix-models/Gemma3-1B", "choices": [{"index": 0, "text": "```json\n{\n \"answer\": \"부산\"\n}\n```", "token_ids": [2717, 3723, 107, 236782, 107, 138, 236775, 14433, 1083, 623, 184764, 236775, 107, 236783, 107, 2717, 106], "finish_reason": "stop"}], "usage": {"prompt_tokens": 8183, "completion_tokens": 17, "total_tokens": 8200}}, "answer_correct": false, "format_correct": false, "elapsed_seconds": 2.009278007899411} diff --git a/opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-rngd-atom.log b/opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-rngd-atom.log new file mode 100644 index 00000000..e287ebe7 --- /dev/null +++ b/opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-rngd-atom.log @@ -0,0 +1,37 @@ +ATOM_BINDING {"visible_count": 1, "physical": 1, "opened": ["/dev/rbln1"]} +arithmetic-0-0 False False +passkey-0-0 False False +korean_table-0-0 False False +arithmetic-1-0 False False +passkey-1-0 False False +korean_table-1-0 False False +arithmetic-2-0 False False +passkey-2-0 False False +korean_table-2-0 False False +arithmetic-3-0 False False +passkey-3-0 False False +korean_table-3-0 False False +arithmetic-0-2048 False False +passkey-0-2048 False False +korean_table-0-2048 False False +arithmetic-1-2048 False False +passkey-1-2048 False False +korean_table-1-2048 False False +arithmetic-2-2048 False False +passkey-2-2048 False False +korean_table-2-2048 False False +arithmetic-3-2048 False False +passkey-3-2048 False False +korean_table-3-2048 False False +arithmetic-0-8192 False False +passkey-0-8192 False False +korean_table-0-8192 False False +arithmetic-1-8192 False False +passkey-1-8192 False False +korean_table-1-8192 False False +arithmetic-2-8192 False False +passkey-2-8192 False False +korean_table-2-8192 False False +arithmetic-3-8192 False False +passkey-3-8192 False False +korean_table-3-8192 False False diff --git a/opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-rngd-atom.producer.jsonl b/opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-rngd-atom.producer.jsonl new file mode 100644 index 00000000..2e678177 --- /dev/null +++ b/opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-rngd-atom.producer.jsonl @@ -0,0 +1,36 @@ +{"wire_id": 0, "start_s": 765340.752833405, "end_s": 765342.361593402} +{"wire_id": 1, "start_s": 765342.562891989, "end_s": 765344.129295412} +{"wire_id": 2, "start_s": 765344.383790441, "end_s": 765345.926818701} +{"wire_id": 3, "start_s": 765346.117121668, "end_s": 765347.675887086} +{"wire_id": 4, "start_s": 765347.846569546, "end_s": 765349.411975671} +{"wire_id": 5, "start_s": 765349.64040735, "end_s": 765351.2042113} +{"wire_id": 6, "start_s": 765351.400314635, "end_s": 765352.959378077} +{"wire_id": 7, "start_s": 765353.143363661, "end_s": 765354.709845499} +{"wire_id": 8, "start_s": 765354.927922674, "end_s": 765356.490186321} +{"wire_id": 9, "start_s": 765356.687235955, "end_s": 765358.249814125} +{"wire_id": 10, "start_s": 765358.436493906, "end_s": 765360.020111918} +{"wire_id": 11, "start_s": 765360.250372575, "end_s": 765361.813592373} +{"wire_id": 12, "start_s": 765362.001979761, "end_s": 765363.63109901} +{"wire_id": 13, "start_s": 765363.859622421, "end_s": 765365.437358235} +{"wire_id": 14, "start_s": 765365.723864237, "end_s": 765367.298855615} +{"wire_id": 15, "start_s": 765367.424889868, "end_s": 765368.998565123} +{"wire_id": 16, "start_s": 765369.233920962, "end_s": 765370.81383167} +{"wire_id": 17, "start_s": 765371.117690276, "end_s": 765372.697017079} +{"wire_id": 18, "start_s": 765372.969031716, "end_s": 765374.549604363} +{"wire_id": 19, "start_s": 765374.776845261, "end_s": 765376.346591589} +{"wire_id": 20, "start_s": 765376.62491273, "end_s": 765378.207728479} +{"wire_id": 21, "start_s": 765378.480830048, "end_s": 765380.054477296} +{"wire_id": 22, "start_s": 765380.319558574, "end_s": 765381.881832839} +{"wire_id": 23, "start_s": 765382.181248512, "end_s": 765383.74680705} +{"wire_id": 24, "start_s": 765384.023430784, "end_s": 765385.732858226} +{"wire_id": 25, "start_s": 765386.147160081, "end_s": 765387.777602672} +{"wire_id": 26, "start_s": 765388.243808587, "end_s": 765389.86937431} +{"wire_id": 27, "start_s": 765390.256338591, "end_s": 765391.885966965} +{"wire_id": 28, "start_s": 765392.223021655, "end_s": 765393.84165539} +{"wire_id": 29, "start_s": 765394.298780325, "end_s": 765395.91144177} +{"wire_id": 30, "start_s": 765396.315891277, "end_s": 765397.93127917} +{"wire_id": 31, "start_s": 765398.278289991, "end_s": 765399.884512323} +{"wire_id": 32, "start_s": 765400.317782739, "end_s": 765401.928910051} +{"wire_id": 33, "start_s": 765402.338907894, "end_s": 765403.918763573} +{"wire_id": 34, "start_s": 765404.276836955, "end_s": 765405.912470594} +{"wire_id": 35, "start_s": 765406.35862692, "end_s": 765407.966054457} diff --git a/opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-rngd-atom.run.json b/opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-rngd-atom.run.json new file mode 100644 index 00000000..d136c7b9 --- /dev/null +++ b/opt_prime/examples/pd/http_matrix/evidence263/quality263-gemma-rngd-atom.run.json @@ -0,0 +1,16 @@ +{ + "stage": "complete", + "route": "rngd-atom", + "model": "unsloth/gemma-3-1b-it", + "runtime_commit": "3e16c33", + "torch": "2.11.0+cpu", + "tokenizer_inputs_equal": true, + "cases": 36, + "context": 9216, + "batch_size": 1, + "kv_blocks": 3, + "cache": false, + "producer_width": 8192, + "answer_correct": 0, + "format_correct": 0 +} diff --git a/opt_prime/examples/pd/http_matrix/evidence263/quality263-pod-after-cleanup.json b/opt_prime/examples/pd/http_matrix/evidence263/quality263-pod-after-cleanup.json new file mode 100644 index 00000000..93fcd81a --- /dev/null +++ b/opt_prime/examples/pd/http_matrix/evidence263/quality263-pod-after-cleanup.json @@ -0,0 +1,5 @@ +{ + "pod_absent": true, + "checked_at": "2026-10-23T21:23:11.908867+00:00", + "owned_pod": "quality263" +} diff --git a/opt_prime/examples/pd/http_matrix/evidence263/quality263-summary.json b/opt_prime/examples/pd/http_matrix/evidence263/quality263-summary.json index c9d59007..5912a191 100644 --- a/opt_prime/examples/pd/http_matrix/evidence263/quality263-summary.json +++ b/opt_prime/examples/pd/http_matrix/evidence263/quality263-summary.json @@ -1417,7 +1417,193 @@ ] }, "rngd-atom": { - "stage": "not_collected" + "stage": "complete", + "cases": 36, + "answer_correct": 0, + "format_correct": 0, + "length_limited": 0, + "wrong_cases": [ + { + "case": "arithmetic-0-0", + "expected": "96", + "text": "```json\n{\n \"answer\": 96\n}\n```" + }, + { + "case": "passkey-0-0", + "expected": "34HYL6W8", + "text": "```json\n{\n \"answer\": \"34HYL6W8\"\n}\n```" + }, + { + "case": "korean_table-0-0", + "expected": "춘천", + "text": "```json\n{\n \"answer\": \"춘천\"\n}\n```" + }, + { + "case": "arithmetic-1-0", + "expected": "83", + "text": "```json\n{\n \"answer\": 83\n}\n```" + }, + { + "case": "passkey-1-0", + "expected": "CNBKEFR7", + "text": "```json\n{\n \"answer\": \"CNBKEFR7\"\n}\n```" + }, + { + "case": "korean_table-1-0", + "expected": "목포", + "text": "```json\n{\n \"answer\": \"목포\"\n}\n```" + }, + { + "case": "arithmetic-2-0", + "expected": "43", + "text": "```json\n{\n \"answer\": 33\n}\n```" + }, + { + "case": "passkey-2-0", + "expected": "WMLLRJUL", + "text": "```json\n{\n \"answer\": \"WMLLRJUL\"\n}\n```" + }, + { + "case": "korean_table-2-0", + "expected": "전주", + "text": "```json\n{\n \"answer\": \"전주\"\n}\n```" + }, + { + "case": "arithmetic-3-0", + "expected": "63", + "text": "```json\n{\n \"answer\": 63\n}\n```" + }, + { + "case": "passkey-3-0", + "expected": "SBYZEGE7", + "text": "```json\n{\n \"answer\": \"SBYZEGE7\"\n}\n```" + }, + { + "case": "korean_table-3-0", + "expected": "속초", + "text": "```json\n{\n \"answer\": \"부산\"\n}\n```" + }, + { + "case": "arithmetic-0-2048", + "expected": "96", + "text": "```json\n{\n \"answer\": 96\n}\n```" + }, + { + "case": "passkey-0-2048", + "expected": "34HYL6W8", + "text": "```json\n{\n \"answer\": \"34HYL6W8\"\n}\n```" + }, + { + "case": "korean_table-0-2048", + "expected": "춘천", + "text": "```json\nnull\n```" + }, + { + "case": "arithmetic-1-2048", + "expected": "83", + "text": "```json\n{\n \"answer\": 83\n}\n```" + }, + { + "case": "passkey-1-2048", + "expected": "CNBKEFR7", + "text": "```json\n{\n \"answer\": \"CNBKEFR7\"\n}\n```" + }, + { + "case": "korean_table-1-2048", + "expected": "목포", + "text": "```json\n{\n \"answer\": \"목포\"\n}\n```" + }, + { + "case": "arithmetic-2-2048", + "expected": "43", + "text": "```json\n{\n \"answer\": 33\n}\n```" + }, + { + "case": "passkey-2-2048", + "expected": "WMLLRJUL", + "text": "```json\n{\n \"answer\": \"WMLLRJUL\"\n}\n```\n" + }, + { + "case": "korean_table-2-2048", + "expected": "전주", + "text": "```json\n{\n \"answer\": \"전주\"\n}\n```" + }, + { + "case": "arithmetic-3-2048", + "expected": "63", + "text": "```json\n{\n \"answer\": 63\n}\n```" + }, + { + "case": "passkey-3-2048", + "expected": "SBYZEGE7", + "text": "```json\n{\n \"answer\": \"SBYZEGE7\"\n}\n```" + }, + { + "case": "korean_table-3-2048", + "expected": "속초", + "text": "```json\n{\n \"answer\": \"속초\"\n}\n```" + }, + { + "case": "arithmetic-0-8192", + "expected": "96", + "text": "```json\n{\n \"answer\": \"96\"\n}\n```" + }, + { + "case": "passkey-0-8192", + "expected": "34HYL6W8", + "text": "```json\n{\n \"answer\": \"34HYL6W8\"\n}\n```" + }, + { + "case": "korean_table-0-8192", + "expected": "춘천", + "text": "```json\n{\n \"answer\": \"부산\"\n}\n```" + }, + { + "case": "arithmetic-1-8192", + "expected": "83", + "text": "```json\n{\"answer\": \"83\"}\n```" + }, + { + "case": "passkey-1-8192", + "expected": "CNBKEFR7", + "text": "```json\n{\n \"answer\": \"CNBKEFR7\"\n}\n```\n" + }, + { + "case": "korean_table-1-8192", + "expected": "목포", + "text": "```json\n{\n \"answer\": \"목포\"\n}\n```" + }, + { + "case": "arithmetic-2-8192", + "expected": "43", + "text": "```json\n{\"answer\":\"33\"}\n```" + }, + { + "case": "passkey-2-8192", + "expected": "WMLLRJUL", + "text": "```json\n{\n \"answer\": \"WMLLRJUL\"\n}\n```" + }, + { + "case": "korean_table-2-8192", + "expected": "전주", + "text": "```json\n{\n \"answer\": \"전주\"\n}\n```" + }, + { + "case": "arithmetic-3-8192", + "expected": "63", + "text": "```json\n{\"answer\": \"63\"}\n```" + }, + { + "case": "passkey-3-8192", + "expected": "SBYZEGE7", + "text": "```json\n{\n \"answer\": \"SBYZEGE7\"\n}\n```" + }, + { + "case": "korean_table-3-8192", + "expected": "속초", + "text": "```json\n{\n \"answer\": \"부산\"\n}\n```" + } + ] } }, "pairs": { @@ -1526,6 +1712,55 @@ } } }, + "native-gpu→rngd-atom": { + "cases": 36, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 33, + "changes": [ + { + "case": "arithmetic-2-0", + "baseline_correct": false, + "candidate_correct": false, + "baseline_text": "```json\n{\n \"answer\": 33\n}\n```\n", + "candidate_text": "```json\n{\n \"answer\": 33\n}\n```" + }, + { + "case": "arithmetic-1-8192", + "baseline_correct": false, + "candidate_correct": false, + "baseline_text": "```json\n{\n \"answer\": \"83\"\n}\n```", + "candidate_text": "```json\n{\"answer\": \"83\"}\n```" + }, + { + "case": "arithmetic-2-8192", + "baseline_correct": false, + "candidate_correct": false, + "baseline_text": "```json\n{\"answer\": \"33\"}\n```", + "candidate_text": "```json\n{\"answer\":\"33\"}\n```" + } + ], + "by_context": { + "0": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 11 + }, + "2048": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + }, + "8192": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 10 + } + } + }, "rngd-gpu→native-atom": { "cases": 36, "correct_to_wrong": 0, @@ -1567,6 +1802,75 @@ "token_equal": 11 } } + }, + "rngd-gpu→rngd-atom": { + "cases": 36, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 34, + "changes": [ + { + "case": "korean_table-0-2048", + "baseline_correct": false, + "candidate_correct": false, + "baseline_text": "```json\n{\n \"answer\": \"춘천\"\n}\n```", + "candidate_text": "```json\nnull\n```" + }, + { + "case": "arithmetic-2-8192", + "baseline_correct": false, + "candidate_correct": false, + "baseline_text": "```json\n{\n \"answer\": \"33\"\n}\n```", + "candidate_text": "```json\n{\"answer\":\"33\"}\n```" + } + ], + "by_context": { + "0": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + }, + "2048": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 11 + }, + "8192": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 11 + } + } + }, + "native-atom→rngd-atom": { + "cases": 36, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 36, + "changes": [], + "by_context": { + "0": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + }, + "2048": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + }, + "8192": { + "cases": 12, + "correct_to_wrong": 0, + "wrong_to_correct": 0, + "token_equal": 12 + } + } } } } diff --git a/opt_prime/examples/pd/http_matrix/experiments/2026-10-24-heldout-quality263.md b/opt_prime/examples/pd/http_matrix/experiments/2026-10-24-heldout-quality263.md index 4ff585c7..67b35a3a 100644 --- a/opt_prime/examples/pd/http_matrix/experiments/2026-10-24-heldout-quality263.md +++ b/opt_prime/examples/pd/http_matrix/experiments/2026-10-24-heldout-quality263.md @@ -84,3 +84,18 @@ Qwen3-4B allfour routes completed: each36/36 structured answers and36/36 format, Compare all six unordered route pairs, not only pairs anchored on nativeGPU. This exposes a Qwen0.6 counterexample: passkey-3-2048 is correct`SBYZEGE7` on RNGD→GPU, but nativeATOM and RNGD→ATOM emit`SYBZEGE7`. Thus each ATOM route has one correct→wrong transition relative to RNGD→GPU, despite no such transition relative to nativeGPU (which also answered this case incorrectly). Do not claim universal cross-route quality equivalence or infer the failing operator without matched-KV controls. KONI allfour routes completed:31/36 structured answers each; nativeGPU/nativeATOM36/36 exacttokens; RNGD→GPU/RNGD→ATOM36/36 exacttokens. Native vsRNGD pair35/36; the differing text only changes JSON whitespace. Gemma nativeATOM completed with code-fenced responses (strictscore0); RNGD→ATOM remains running. + +## Final completion + +All576 responses (4models×4routes×36cases) completed with normal EOS and no length-capped outputs or HTTP/inference failures. Frozen fixture hashes and all returned case/input hashes verified; exact parsed-answer scores recomputed from raw text. OwnedPod quality263 deletion independently confirmed absent. + +| Model | nativeGPU | RNGD→GPU | nativeATOM | RNGD→ATOM | +|---|---:|---:|---:|---:| +| Qwen3-0.6B |32/36|34/36|33/36|33/36| +| Qwen3-4B |36/36|36/36|36/36|36/36| +| KONI-8B |31/36|31/36|31/36|31/36| +| Gemma3-1B |0/36|0/36|0/36|0/36| + +These are strict structured-answer adherence counts, not unrestricted semantic accuracy. Gemma emits Markdown code fences on every route. Its final ATOM native/PD pair is token-exact36/36; GPU→ATOM comparison still changes content: korean_table-0-2048 yields correct`춘천` on RNGD→GPU but`null` on RNGD→ATOM and nativeATOM. This is in addition to the Qwen0.6 passkey counterexample. Neither counterexample can be blamed on a particular operator or transport without matched-KV controls. + +NativeATOM/RNGD→ATOM exact pairs: Q06 36/36, Q4 36/36, KONI35/36 (JSON whitespace only), Gemma36/36. All six route-pair comparisons, per-context counts and changed raw texts are in quality263-summary.json. No universal quality-equivalence or performance-speedup conclusion. Numerical issues#233/#256/#261 and actual within-request streaming#265 remain separate work. From 4d738b6d5c1106b32685cd0befd10f64c8eb1a0e Mon Sep 17 00:00:00 2001 From: gspark Date: Sat, 24 Oct 2026 08:05:31 +0900 Subject: [PATCH 7/7] docs: separate held-out content errors from format failures (#263) --- .../pd/http_matrix/.research/events.jsonl | 1 + opt_prime/examples/pd/http_matrix/STATUS.md | 2 + .../evidence263/analyze_quality263_content.py | 50 ++ .../quality263-content-diagnostic.json | 609 ++++++++++++++++++ .../2026-10-24-heldout-quality263.md | 6 + 5 files changed, 668 insertions(+) create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/analyze_quality263_content.py create mode 100644 opt_prime/examples/pd/http_matrix/evidence263/quality263-content-diagnostic.json diff --git a/opt_prime/examples/pd/http_matrix/.research/events.jsonl b/opt_prime/examples/pd/http_matrix/.research/events.jsonl index 98d83845..61fb3f9b 100644 --- a/opt_prime/examples/pd/http_matrix/.research/events.jsonl +++ b/opt_prime/examples/pd/http_matrix/.research/events.jsonl @@ -9,3 +9,4 @@ {"id": "2919d0420ffc49ceacc9aa8982bf4859", "ts": "2026-10-24T04:48:11+09:00", "task": "Four-model serving matrix (aicomp #201)", "result": "All4 live GPU lifecycle runs completed passed=true, including KONI/Gemma: cancellation/reuse, producer death503/readiness, slab unlink and replacement exact32-token controls. Started separate512-token concurrent-decode isolation suite after prior queue terminal and device-idle checks.", "next": "Publish remaining lifecycle evidence; verify currently decoding requests survive producer failure across4models", "state": "active", "done": false, "blocker": "", "metrics": {}, "params": {}, "artifacts": [{"path": "analysis/serving-matrix-201/local201-koni-faults.json", "size": 1003, "sha256": "5ef1011fb2c2a4c6"}, {"path": "analysis/serving-matrix-201/local201-gemma-faults.json", "size": 1016, "sha256": "e707fb1f52adbb1a"}], "experiments": ["experiments/2026-10-24-live-gpu-fault201.md"], "labels": ["benchmark"], "review": false, "todos": [], "note": "", "agent": "codex", "host": "xeon-max", "branch": "main", "commit": "351d39d-dirty"} {"id": "eb53d939aca44b85ab4b6fd602754651", "ts": "2026-10-24T04:58:02+09:00", "task": "Four-model serving matrix (aicomp #201)", "result": "All4 GPU concurrent-decode isolation tests passed512exact tokens after producer killed at token2, plus cancellation/503/slab-unlink/replacement. KONI paddedGPU control completed: changes output but does NOT reproduce short RNGD output; padding-only explanation rejected.", "next": "Publish all4isolation evidence; capture short KONI KV for identical-prefix numerical analysis", "state": "active", "done": false, "blocker": "", "metrics": {}, "params": {}, "artifacts": [{"path": "analysis/serving-matrix-201/koni-short-padding-diagnostic.json", "size": 5903, "sha256": "f67039817a78849a"}], "experiments": ["experiments/2026-10-24-live-gpu-fault201.md"], "labels": ["benchmark"], "review": false, "todos": [], "note": "", "agent": "codex", "host": "xeon-max", "branch": "main", "commit": "351d39d-dirty"} {"id": "ce47a4a232ae4f6dbb19f8738f8b2364", "ts": "2026-10-24T06:29:28+09:00", "task": "Structured answer compatibility (aicomp #263)", "result": "Completed all576 responses, normal EOS and no inference failures. Qwen3-4B allfour routes36/36 answers and token-exact; Q06 and Gemma retain cross-route content counterexamples. Final published artifact auditor reproduces summary exactly; owned Pod absent.", "next": "Review PR264; investigate content differences with matched-KV controls; complete source-side streaming in #265", "state": "active", "done": false, "blocker": "", "metrics": {}, "params": {"models": "4", "routes": "4", "cases_per_model": "36"}, "artifacts": ["evidence263/quality263-summary.json"], "experiments": ["experiments/2026-10-24-heldout-quality263.md"], "labels": ["experiment", "rngd"], "review": true, "todos": [], "note": "", "agent": "codex", "host": "xeon-max", "branch": "main", "commit": "351d39d-dirty"} +{"id": "57487a847f5f468cbfb29adf40acbdb9", "ts": "2026-10-24T08:05:12+09:00", "task": "Held-out answer quality (aicomp #263)", "result": "Post-hoc content diagnostic on unchanged576 responses: Gemma29/30/29/29 correct after outer-fence/arithmetic-int relaxation vs frozen primary0/0/0/0. Other models unchanged; one Gemma cross-route content difference remains. All16 files match PR264, self-checks and deterministic rerun pass; primary criteria not changed.", "next": "Publish diagnostic as exploratory in PR264; preserve native-parity failures; continue active long ATOM timing64332", "state": "active", "done": false, "blocker": "", "metrics": {}, "params": {}, "artifacts": ["evidence263/quality263-content-diagnostic.json"], "experiments": ["experiments/2026-10-24-heldout-quality263.md"], "labels": ["analysis", "rngd"], "review": true, "todos": [], "note": "", "agent": "codex", "host": "analysis host", "branch": "main", "commit": "351d39d-dirty"} diff --git a/opt_prime/examples/pd/http_matrix/STATUS.md b/opt_prime/examples/pd/http_matrix/STATUS.md index a8237d8e..d6484b8c 100644 --- a/opt_prime/examples/pd/http_matrix/STATUS.md +++ b/opt_prime/examples/pd/http_matrix/STATUS.md @@ -63,3 +63,5 @@ Allfour live GPU isolation runs pass: active512-token decode survives other-requ ## Final held-out evaluation (#263) All 576 responses completed normally. Qwen3-4B: 36/36 structured answers and exact tokens on all four routes. Qwen3-0.6B and Gemma retain cross-route content counterexamples; all six route pairs are audited. Final published artifact auditor reproduces the summary exactly. Owned evaluation Pod absent. See evidence263 and the heldout-quality263 experiment; numerical compatibility and source streaming remain open. + +Post-hoc #263 content diagnostic (primary unchanged): Gemma29/30/29/29 of36 correct after only outer-fence/arithmetic-int relaxation, vs frozen primary0/0/0/0. Six or seven content errors and one cross-route answer difference remain. Other model scores unchanged. All16 raw JSONL inputs byte-identical to published evidence; analyzer self-checks/reproduction pass. Exploratory, not an acceptance-rule change. diff --git a/opt_prime/examples/pd/http_matrix/evidence263/analyze_quality263_content.py b/opt_prime/examples/pd/http_matrix/evidence263/analyze_quality263_content.py new file mode 100644 index 00000000..71c40841 --- /dev/null +++ b/opt_prime/examples/pd/http_matrix/evidence263/analyze_quality263_content.py @@ -0,0 +1,50 @@ +"""Post-hoc diagnostic only: preserve frozen primary scores, separate wrapper/type failures.""" +import json,re,sys,itertools +from pathlib import Path + +def content(row): + text=row['text'].strip();fenced=False + match=re.fullmatch(r'```(?:json)?[ \t]*\n(.*?)\n```',text,flags=re.S) + if match:text=match[1];fenced=True + def unique(pairs): + if len(dict(pairs))!=len(pairs):raise ValueError("duplicate keys") + return dict(pairs) + try:obj=json.loads(text,object_pairs_hook=unique) + except (ValueError,TypeError):return None,fenced,False + if not isinstance(obj,dict) or set(obj)!={'answer'}:return None,fenced,False + answer=obj['answer'];numeric=type(answer) is int and row['case'].startswith('arithmetic-') + if numeric:answer=str(answer) + if not isinstance(answer,str):answer=None + return answer,fenced,numeric + +def check(): + def row(text,case='arithmetic-0-0'):return dict(text=text,case=case) + assert content(row('```json\n{"answer": 96}\n```'))==('96',True,True) + assert content(row('{"answer": true}'))[0] is None + assert content(row('{"answer": 96.0}'))[0] is None + assert content(row('{"answer": 96}','passkey-0-0'))[0] is None + assert content(row('prefix ```json\n{"answer":"96"}\n```'))[0] is None + assert content(row('{"answer":"096"}'))[0]=='096' + assert content(row('{"answer":null}'))[0] is None + assert content(row('{"answer":"wrong","answer":"96"}'))[0] is None + assert content(row('{"answer":"96","extra":1}'))[0] is None + +check() +if __name__=='__main__': + root=Path(sys.argv[1]);out={'scope':'Post-hoc exploratory diagnostic. Frozen primary scores unchanged; this is not a replacement acceptance criterion. Only an outer JSON/unlabelled fence and integer-to-string arithmetic answers are relaxed.','models':{}} + routes=('native-gpu','rngd-gpu','native-atom','rngd-atom') + for model in ('qwen06b','qwen4b','koni','gemma'): + rows={route:[json.loads(x) for x in (root/f'quality263-{model}-{route}.jsonl').read_text().splitlines()] for route in routes} + assert all(len(v)==36 and len({r['case'] for r in v})==36 for v in rows.values()) + stats={};parsed={} + for route,records in rows.items(): + parsed[route]={r['case']:content(r)[0] for r in records} + stats[route]=dict(cases=36,primary_correct=sum(r['answer_correct'] for r in records),content_correct=sum(content(r)[0]==r['expected'] for r in records),outer_fence=sum(content(r)[1] for r in records),arithmetic_integer=sum(content(r)[2] for r in records),content_errors=[dict(case=r['case'],expected=r['expected'],answer=content(r)[0],text=r['text']) for r in records if content(r)[0]!=r['expected']]) + pairs={} + for a,b in itertools.combinations(routes,2): + aa={r['case']:r for r in rows[a]};bb={r['case']:r for r in rows[b]} + assert aa.keys()==bb.keys() + assert all(aa[k]['input_sha256']==bb[k]['input_sha256'] and aa[k]['expected']==bb[k]['expected'] for k in aa) + pairs[a+' vs '+b]=[dict(case=k,left=parsed[a][k],right=parsed[b][k],expected=aa[k]['expected']) for k in aa if parsed[a][k]!=parsed[b][k]] + out['models'][model]=dict(routes=stats,content_differences=pairs) + print(json.dumps(out,ensure_ascii=False,indent=2)) diff --git a/opt_prime/examples/pd/http_matrix/evidence263/quality263-content-diagnostic.json b/opt_prime/examples/pd/http_matrix/evidence263/quality263-content-diagnostic.json new file mode 100644 index 00000000..95258fb4 --- /dev/null +++ b/opt_prime/examples/pd/http_matrix/evidence263/quality263-content-diagnostic.json @@ -0,0 +1,609 @@ +{ + "scope": "Post-hoc exploratory diagnostic. Frozen primary scores unchanged; this is not a replacement acceptance criterion. Only an outer JSON/unlabelled fence and integer-to-string arithmetic answers are relaxed.", + "models": { + "qwen06b": { + "routes": { + "native-gpu": { + "cases": 36, + "primary_correct": 32, + "content_correct": 32, + "outer_fence": 0, + "arithmetic_integer": 0, + "content_errors": [ + { + "case": "korean_table-1-0", + "expected": "목포", + "answer": "부산", + "text": "{\n \"answer\": \"부산\"\n}" + }, + { + "case": "passkey-3-0", + "expected": "SBYZEGE7", + "answer": "SYBZEGE7", + "text": "{\n \"answer\": \"SYBZEGE7\"\n}" + }, + { + "case": "passkey-3-2048", + "expected": "SBYZEGE7", + "answer": "SYBZEGE7", + "text": "{\n \"answer\": \"SYBZEGE7\"\n}" + }, + { + "case": "passkey-3-8192", + "expected": "SBYZEGE7", + "answer": "SYBZEGE7", + "text": "{\n \"answer\": \"SYBZEGE7\"\n}" + } + ] + }, + "rngd-gpu": { + "cases": 36, + "primary_correct": 34, + "content_correct": 34, + "outer_fence": 0, + "arithmetic_integer": 0, + "content_errors": [ + { + "case": "passkey-3-0", + "expected": "SBYZEGE7", + "answer": "SYBZEGE7", + "text": "{\n \"answer\": \"SYBZEGE7\"\n}" + }, + { + "case": "passkey-3-8192", + "expected": "SBYZEGE7", + "answer": "SYBZEGE7", + "text": "{\n \"answer\": \"SYBZEGE7\"\n}" + } + ] + }, + "native-atom": { + "cases": 36, + "primary_correct": 33, + "content_correct": 33, + "outer_fence": 0, + "arithmetic_integer": 0, + "content_errors": [ + { + "case": "passkey-3-0", + "expected": "SBYZEGE7", + "answer": "SYBZEGE7", + "text": "{\n \"answer\": \"SYBZEGE7\"\n}" + }, + { + "case": "passkey-3-2048", + "expected": "SBYZEGE7", + "answer": "SYBZEGE7", + "text": "{\n \"answer\": \"SYBZEGE7\"\n}" + }, + { + "case": "passkey-3-8192", + "expected": "SBYZEGE7", + "answer": "SYBZEGE7", + "text": "{\n \"answer\": \"SYBZEGE7\"\n}" + } + ] + }, + "rngd-atom": { + "cases": 36, + "primary_correct": 33, + "content_correct": 33, + "outer_fence": 0, + "arithmetic_integer": 0, + "content_errors": [ + { + "case": "passkey-3-0", + "expected": "SBYZEGE7", + "answer": "SYBZEGE7", + "text": "{\n \"answer\": \"SYBZEGE7\"\n}" + }, + { + "case": "passkey-3-2048", + "expected": "SBYZEGE7", + "answer": "SYBZEGE7", + "text": "{\n \"answer\": \"SYBZEGE7\"\n}" + }, + { + "case": "passkey-3-8192", + "expected": "SBYZEGE7", + "answer": "SYBZEGE7", + "text": "{\n \"answer\": \"SYBZEGE7\"\n}" + } + ] + } + }, + "content_differences": { + "native-gpu vs rngd-gpu": [ + { + "case": "korean_table-1-0", + "left": "부산", + "right": "목포", + "expected": "목포" + }, + { + "case": "passkey-3-2048", + "left": "SYBZEGE7", + "right": "SBYZEGE7", + "expected": "SBYZEGE7" + } + ], + "native-gpu vs native-atom": [ + { + "case": "korean_table-1-0", + "left": "부산", + "right": "목포", + "expected": "목포" + } + ], + "native-gpu vs rngd-atom": [ + { + "case": "korean_table-1-0", + "left": "부산", + "right": "목포", + "expected": "목포" + } + ], + "rngd-gpu vs native-atom": [ + { + "case": "passkey-3-2048", + "left": "SBYZEGE7", + "right": "SYBZEGE7", + "expected": "SBYZEGE7" + } + ], + "rngd-gpu vs rngd-atom": [ + { + "case": "passkey-3-2048", + "left": "SBYZEGE7", + "right": "SYBZEGE7", + "expected": "SBYZEGE7" + } + ], + "native-atom vs rngd-atom": [] + } + }, + "qwen4b": { + "routes": { + "native-gpu": { + "cases": 36, + "primary_correct": 36, + "content_correct": 36, + "outer_fence": 0, + "arithmetic_integer": 0, + "content_errors": [] + }, + "rngd-gpu": { + "cases": 36, + "primary_correct": 36, + "content_correct": 36, + "outer_fence": 0, + "arithmetic_integer": 0, + "content_errors": [] + }, + "native-atom": { + "cases": 36, + "primary_correct": 36, + "content_correct": 36, + "outer_fence": 0, + "arithmetic_integer": 0, + "content_errors": [] + }, + "rngd-atom": { + "cases": 36, + "primary_correct": 36, + "content_correct": 36, + "outer_fence": 0, + "arithmetic_integer": 0, + "content_errors": [] + } + }, + "content_differences": { + "native-gpu vs rngd-gpu": [], + "native-gpu vs native-atom": [], + "native-gpu vs rngd-atom": [], + "rngd-gpu vs native-atom": [], + "rngd-gpu vs rngd-atom": [], + "native-atom vs rngd-atom": [] + } + }, + "koni": { + "routes": { + "native-gpu": { + "cases": 36, + "primary_correct": 31, + "content_correct": 31, + "outer_fence": 0, + "arithmetic_integer": 0, + "content_errors": [ + { + "case": "korean_table-0-0", + "expected": "춘천", + "answer": "부산", + "text": "{\"answer\": \"부산\"}" + }, + { + "case": "arithmetic-1-0", + "expected": "83", + "answer": "48 + 35 = 83", + "text": "{\n \"answer\": \"48 + 35 = 83\"\n}" + }, + { + "case": "arithmetic-2-0", + "expected": "43", + "answer": "The sum of 32 and 11 is 43.", + "text": "{\n \"answer\": \"The sum of 32 and 11 is 43.\"\n}" + }, + { + "case": "passkey-0-8192", + "expected": "34HYL6W8", + "answer": null, + "text": "34HYL6W8" + }, + { + "case": "passkey-2-8192", + "expected": "WMLLRJUL", + "answer": null, + "text": "WMLLRJUL" + } + ] + }, + "rngd-gpu": { + "cases": 36, + "primary_correct": 31, + "content_correct": 31, + "outer_fence": 0, + "arithmetic_integer": 0, + "content_errors": [ + { + "case": "korean_table-0-0", + "expected": "춘천", + "answer": "부산", + "text": "{\"answer\": \"부산\"}" + }, + { + "case": "arithmetic-1-0", + "expected": "83", + "answer": "48 + 35 = 83", + "text": "{\n \"answer\": \"48 + 35 = 83\"\n}" + }, + { + "case": "arithmetic-2-0", + "expected": "43", + "answer": "The sum of 32 and 11 is 43.", + "text": "{\n \"answer\": \"The sum of 32 and 11 is 43.\"\n}" + }, + { + "case": "passkey-0-8192", + "expected": "34HYL6W8", + "answer": null, + "text": "34HYL6W8" + }, + { + "case": "passkey-2-8192", + "expected": "WMLLRJUL", + "answer": null, + "text": "WMLLRJUL" + } + ] + }, + "native-atom": { + "cases": 36, + "primary_correct": 31, + "content_correct": 31, + "outer_fence": 0, + "arithmetic_integer": 0, + "content_errors": [ + { + "case": "korean_table-0-0", + "expected": "춘천", + "answer": "부산", + "text": "{\"answer\": \"부산\"}" + }, + { + "case": "arithmetic-1-0", + "expected": "83", + "answer": "48 + 35 = 83", + "text": "{\n \"answer\": \"48 + 35 = 83\"\n}" + }, + { + "case": "arithmetic-2-0", + "expected": "43", + "answer": "The sum of 32 and 11 is 43.", + "text": "{\n \"answer\": \"The sum of 32 and 11 is 43.\"\n}" + }, + { + "case": "passkey-0-8192", + "expected": "34HYL6W8", + "answer": null, + "text": "34HYL6W8" + }, + { + "case": "passkey-2-8192", + "expected": "WMLLRJUL", + "answer": null, + "text": "WMLLRJUL" + } + ] + }, + "rngd-atom": { + "cases": 36, + "primary_correct": 31, + "content_correct": 31, + "outer_fence": 0, + "arithmetic_integer": 0, + "content_errors": [ + { + "case": "korean_table-0-0", + "expected": "춘천", + "answer": "부산", + "text": "{\"answer\": \"부산\"}" + }, + { + "case": "arithmetic-1-0", + "expected": "83", + "answer": "48 + 35 = 83", + "text": "{\n \"answer\": \"48 + 35 = 83\"\n}" + }, + { + "case": "arithmetic-2-0", + "expected": "43", + "answer": "The sum of 32 and 11 is 43.", + "text": "{\n \"answer\": \"The sum of 32 and 11 is 43.\"\n}" + }, + { + "case": "passkey-0-8192", + "expected": "34HYL6W8", + "answer": null, + "text": "34HYL6W8" + }, + { + "case": "passkey-2-8192", + "expected": "WMLLRJUL", + "answer": null, + "text": "WMLLRJUL" + } + ] + } + }, + "content_differences": { + "native-gpu vs rngd-gpu": [], + "native-gpu vs native-atom": [], + "native-gpu vs rngd-atom": [], + "rngd-gpu vs native-atom": [], + "rngd-gpu vs rngd-atom": [], + "native-atom vs rngd-atom": [] + } + }, + "gemma": { + "routes": { + "native-gpu": { + "cases": 36, + "primary_correct": 0, + "content_correct": 29, + "outer_fence": 36, + "arithmetic_integer": 8, + "content_errors": [ + { + "case": "arithmetic-2-0", + "expected": "43", + "answer": "33", + "text": "```json\n{\n \"answer\": 33\n}\n```\n" + }, + { + "case": "korean_table-3-0", + "expected": "속초", + "answer": "부산", + "text": "```json\n{\n \"answer\": \"부산\"\n}\n```" + }, + { + "case": "korean_table-0-2048", + "expected": "춘천", + "answer": null, + "text": "```json\nnull\n```" + }, + { + "case": "arithmetic-2-2048", + "expected": "43", + "answer": "33", + "text": "```json\n{\n \"answer\": 33\n}\n```" + }, + { + "case": "korean_table-0-8192", + "expected": "춘천", + "answer": "부산", + "text": "```json\n{\n \"answer\": \"부산\"\n}\n```" + }, + { + "case": "arithmetic-2-8192", + "expected": "43", + "answer": "33", + "text": "```json\n{\"answer\": \"33\"}\n```" + }, + { + "case": "korean_table-3-8192", + "expected": "속초", + "answer": "부산", + "text": "```json\n{\n \"answer\": \"부산\"\n}\n```" + } + ] + }, + "rngd-gpu": { + "cases": 36, + "primary_correct": 0, + "content_correct": 30, + "outer_fence": 36, + "arithmetic_integer": 8, + "content_errors": [ + { + "case": "arithmetic-2-0", + "expected": "43", + "answer": "33", + "text": "```json\n{\n \"answer\": 33\n}\n```" + }, + { + "case": "korean_table-3-0", + "expected": "속초", + "answer": "부산", + "text": "```json\n{\n \"answer\": \"부산\"\n}\n```" + }, + { + "case": "arithmetic-2-2048", + "expected": "43", + "answer": "33", + "text": "```json\n{\n \"answer\": 33\n}\n```" + }, + { + "case": "korean_table-0-8192", + "expected": "춘천", + "answer": "부산", + "text": "```json\n{\n \"answer\": \"부산\"\n}\n```" + }, + { + "case": "arithmetic-2-8192", + "expected": "43", + "answer": "33", + "text": "```json\n{\n \"answer\": \"33\"\n}\n```" + }, + { + "case": "korean_table-3-8192", + "expected": "속초", + "answer": "부산", + "text": "```json\n{\n \"answer\": \"부산\"\n}\n```" + } + ] + }, + "native-atom": { + "cases": 36, + "primary_correct": 0, + "content_correct": 29, + "outer_fence": 36, + "arithmetic_integer": 8, + "content_errors": [ + { + "case": "arithmetic-2-0", + "expected": "43", + "answer": "33", + "text": "```json\n{\n \"answer\": 33\n}\n```" + }, + { + "case": "korean_table-3-0", + "expected": "속초", + "answer": "부산", + "text": "```json\n{\n \"answer\": \"부산\"\n}\n```" + }, + { + "case": "korean_table-0-2048", + "expected": "춘천", + "answer": null, + "text": "```json\nnull\n```" + }, + { + "case": "arithmetic-2-2048", + "expected": "43", + "answer": "33", + "text": "```json\n{\n \"answer\": 33\n}\n```" + }, + { + "case": "korean_table-0-8192", + "expected": "춘천", + "answer": "부산", + "text": "```json\n{\n \"answer\": \"부산\"\n}\n```" + }, + { + "case": "arithmetic-2-8192", + "expected": "43", + "answer": "33", + "text": "```json\n{\"answer\":\"33\"}\n```" + }, + { + "case": "korean_table-3-8192", + "expected": "속초", + "answer": "부산", + "text": "```json\n{\n \"answer\": \"부산\"\n}\n```" + } + ] + }, + "rngd-atom": { + "cases": 36, + "primary_correct": 0, + "content_correct": 29, + "outer_fence": 36, + "arithmetic_integer": 8, + "content_errors": [ + { + "case": "arithmetic-2-0", + "expected": "43", + "answer": "33", + "text": "```json\n{\n \"answer\": 33\n}\n```" + }, + { + "case": "korean_table-3-0", + "expected": "속초", + "answer": "부산", + "text": "```json\n{\n \"answer\": \"부산\"\n}\n```" + }, + { + "case": "korean_table-0-2048", + "expected": "춘천", + "answer": null, + "text": "```json\nnull\n```" + }, + { + "case": "arithmetic-2-2048", + "expected": "43", + "answer": "33", + "text": "```json\n{\n \"answer\": 33\n}\n```" + }, + { + "case": "korean_table-0-8192", + "expected": "춘천", + "answer": "부산", + "text": "```json\n{\n \"answer\": \"부산\"\n}\n```" + }, + { + "case": "arithmetic-2-8192", + "expected": "43", + "answer": "33", + "text": "```json\n{\"answer\":\"33\"}\n```" + }, + { + "case": "korean_table-3-8192", + "expected": "속초", + "answer": "부산", + "text": "```json\n{\n \"answer\": \"부산\"\n}\n```" + } + ] + } + }, + "content_differences": { + "native-gpu vs rngd-gpu": [ + { + "case": "korean_table-0-2048", + "left": null, + "right": "춘천", + "expected": "춘천" + } + ], + "native-gpu vs native-atom": [], + "native-gpu vs rngd-atom": [], + "rngd-gpu vs native-atom": [ + { + "case": "korean_table-0-2048", + "left": "춘천", + "right": null, + "expected": "춘천" + } + ], + "rngd-gpu vs rngd-atom": [ + { + "case": "korean_table-0-2048", + "left": "춘천", + "right": null, + "expected": "춘천" + } + ], + "native-atom vs rngd-atom": [] + } + } + } +} diff --git a/opt_prime/examples/pd/http_matrix/experiments/2026-10-24-heldout-quality263.md b/opt_prime/examples/pd/http_matrix/experiments/2026-10-24-heldout-quality263.md index 67b35a3a..7fd75655 100644 --- a/opt_prime/examples/pd/http_matrix/experiments/2026-10-24-heldout-quality263.md +++ b/opt_prime/examples/pd/http_matrix/experiments/2026-10-24-heldout-quality263.md @@ -99,3 +99,9 @@ All576 responses (4models×4routes×36cases) completed with normal EOS and no le These are strict structured-answer adherence counts, not unrestricted semantic accuracy. Gemma emits Markdown code fences on every route. Its final ATOM native/PD pair is token-exact36/36; GPU→ATOM comparison still changes content: korean_table-0-2048 yields correct`춘천` on RNGD→GPU but`null` on RNGD→ATOM and nativeATOM. This is in addition to the Qwen0.6 passkey counterexample. Neither counterexample can be blamed on a particular operator or transport without matched-KV controls. NativeATOM/RNGD→ATOM exact pairs: Q06 36/36, Q4 36/36, KONI35/36 (JSON whitespace only), Gemma36/36. All six route-pair comparisons, per-context counts and changed raw texts are in quality263-summary.json. No universal quality-equivalence or performance-speedup conclusion. Numerical issues#233/#256/#261 and actual within-request streaming#265 remain separate work. + +## Post-hoc content diagnostic (primary criteria unchanged) + +No new inference. All16 input JSONL files are byte-identical to published PR264 evidence atc28d056. `python3 analysis/serving-matrix-201/analyze_quality263_content.py analysis/serving-matrix-201 > analysis/serving-matrix-201/quality263-content-diagnostic.json` checks all576 frozen records, case/input hashes/expected answers across all six route pairs. Separate exploratory metric removes only a whole outer JSON/unlabelled code fence and accepts integer arithmetic answers as their exact decimal string. No substring matching, float/bool/null coercion, case folding or extra-key acceptance; duplicate JSON keys rejected. Embedded self-checks accept the intended relaxation and reject malformed/ambiguous/over-permissive cases. Rerun output is byte-identical. These rules were selected after seeing outputs and must not replace the frozen primary scores or waive earlier exactness failures. + +Route order: nativeGPU / RNGD-GPU / nativeATOM / RNGD-ATOM. Exploratory content scores outof36: Qwen0.6=32/34/33/33; Qwen4=36/36/36/36; KONI=31/31/31/31; Gemma=29/30/29/29. Gemma primary remains0/0/0/0. All36 Gemma responses per route have outer fences;8 per route additionally use integer arithmetic answers. Thus formatting masks substantial content correctness but also genuine6–7 content errors. Exactly one Gemma case differs in normalized answer between RNGD-GPU and the other routes, the already-retained Korean city lookup; Qwen0.6 retains its prior content differences. KONI/Qwen4 normalized answers match across allsix pairs on these fixtures only. No operator or transport root-cause claim.