docs: record four-model ATOM serving validation and limits

experiments/evidence270/NUMERICS-DIAGNOSTICS.md

@@ -0,0 +1,41 @@
+# Qwen3-4B diagnostic progression and limits
+
+The native B4 parity failure remains unresolved. These isolated exports are diagnostic experiments, not serving fixes or recommended defaults. All use the SDK environment and PR266 prerequisites in QWEN4-LANE-REPRO.md. Shared SDK files and engine code were unchanged.
+
+| Experiment | Result | Limit |
+|---|---|---|
+| Supported output_hidden_states export | Both original token splits reproduce. At all four observed positions, embeddings are lane-equal; first difference at boundary1, after layer0. | Recorded logits differ from the original uninstrumented graph. |
+| First-layer physical KV readback | Sixteen K/V snapshots show identical initialized ranges across all four blocks. Changing only block3 token0 changes1024 K and1024 V elements; restoring the prompt restores equality. | Sampled first-layer stored KV only. Does not identify a compute operator. |
+| First-layer module hooks | Input normalization stays equal; first difference at attention output, including output projection. Prior instrumented logit records match. | Extra graph outputs may change optimization. MLP receives differing inputs, so no MLP-origin conclusion. |
+| Packed attention-internal outputs | Export falls back to jittrace, then DEVICE_GRAPH_CONVERSION failure. | No runtime numerical result; cause of compiler rejection unidentified. |
+| Raw attention-internal decode outputs | Compiles; all observed first-attention internals are lane-equal.2048/index13 split remains,512/index50 split disappears. Later retained boundary11 differs. | Instrumentation materially changes numerical behavior. Not a fix and not proof the original attention kernel is correct or incorrect. |
+
+Compile times: supported hidden153.837882s; module hooks154.000048s; raw attention152.587502s. All owned Pods were cleaned up, including the failed compile attempt. No throughput or semantic-quality gain is claimed.
+
+## Interpretation
+
+Both module-hook and raw-attention exports log two torch.export→jittrace fallbacks; the supported hidden-state export logs none. Fallback alone does not distinguish the two instrumented outcomes. Any further attribution needs matched compiler controls or a smaller isolated replay with the same numerical failure. Do not adjust greedy tie-breaking to hide a changed logit order.
+
+Raw attention diagnostic slots0..10 are embedding, Q/K/V projections, Q/K normalization, rotary Q/K, attention V input, core attention output and output projection. Slots11..36 retain the ordinary hidden-state boundaries. Raw native tensor shapes are preserved only in decode; prefill keeps the original fixed-size hidden-output interface. Layer-module diagnostic slots0..5 have separate labels in their result. Neither layout is a general serving hidden-state API.
+
+## Reproduction
+
+Use a fresh task directory, the compact fixture from QWEN4-LANE-REPRO.md, the existing bootstrap/validation helpers and the recorded SDK versions. Allocate only the owned ATOM. Builders create distinct atom270-artifacts-* cache roots and reject overwrites. Run each compiler and replay in separate Python processes:
+
+```bash
+"$ATOM_PYTHON" build_atom270_hidden.py
+"$ATOM_PYTHON" container_bootstrap270.py diagnose_atom270_hidden.py
+"$ATOM_PYTHON" container_bootstrap270.py diagnose_atom270_firstkv_control.py
+"$ATOM_PYTHON" build_atom270_layerprobe.py
+"$ATOM_PYTHON" container_bootstrap270.py diagnose_atom270_layerprobe.py
+"$ATOM_PYTHON" build_atom270_attentionraw.py
+"$ATOM_PYTHON" container_bootstrap270.py diagnose_atom270_attentionraw.py
+```
+
+Compact audits and source hashes are included. Raw token/probe arrays and failed compile log remain outside Git, linked by full hashes and research event IDs in diagnostic-evidence-manifest.json. Remote archive synchronization is unverified. Packaging does not constitute a new hardware run, nor a full repository test suite.
+
+## Required next evidence
+
+1. Confirm the original numerical failure under a controlled tracing/compiler comparison without changing observation outputs.
+2. If using isolated replay, preserve real inputs/KV and establish the original failure before changing an operator.
+3. Validate any candidate on both fixtures, all lane placements, free generation and held-out quality, then matched four-model serving and fault tests. Keep B1 defaults and B4 parity failure until these checks pass.

experiments/evidence270/QWEN4-LANE-REPRO.md

@@ -0,0 +1,38 @@
+# Qwen3-4B B4 lane-dependent logits
+
+This reproduction is for engine/SDK diagnosis, not a correctness waiver or a serving fix.
+
+## Observed failure
+
+Qwen/Qwen3-4B, ATOM B4/TP1, five KV blocks, context9216. At input2048/output index13, forced identical history gives lane0 token369 with logits(369,264)=(29.75,29.625); lanes1–3 choose264 with(29.625,29.625). KV block permutation and fixing prefill batch_idx0 do not remove the split. Same-Pod ABBA completes128 cases with identical tokens and recorded logits across arms. Four bootstrap records verify the same assigned physical ATOM.
+
+Earlier direct SDK replay bypasses AtomBatcher and HTTP entirely and reproduces this split with identical per-lane decode tokens/positions. It also reproduces the512-input/index50 split (4829 versus1033). Native SDK output is bfloat16. This narrows reproduction to the SDK/compiled decode path; the responsible operator and preferred numerical result remain unidentified.
+
+## Environment and prerequisites
+
+Measured runtime is the PR266 stack (eaf7daa219a6cea68db4ec5fc08850edc848b271), rebel-compiler0.11.0, optimum-rbln0.11.0.post1, torch2.11.0+cpu, Transformers5.8.1. Use the previously compiled Qwen3-4B B4/k5/context9216 artifact. The scripts load from an atom270-artifacts directory alongside themselves. Provide the existing container_bootstrap270.py and atom_server270_http.py validation helper through the task harness/PYTHONPATH. Allocate one ATOM; no RNGD is needed. Do not reuse an unrelated workload or regroup devices. This package is not standalone against main.
+
+The compact fixture contains only the exact synthetic prompt IDs and reference tokens consumed by these scripts. It is an exact subset of the preserved native HTTP result; the source hash is in samecard-evidence-manifest.json. Reference tokens are replay inputs, not an independent quality oracle.
+
+## Run
+
+Copy the source scripts and fixture to a fresh task directory with the prerequisites above. The measured scripts retain their original input filename:
+
+```bash
+cp qwen4-lane-fixtures.json atom270-qwen4b-tp1-b4-native.json
+python3 selfcheck_atom270_direct.py
+"$ATOM_PYTHON" container_bootstrap270.py diagnose_atom270_direct.py
+for arm in a1 b1 b2 a2; do
+  "$ATOM_PYTHON" container_bootstrap270.py diagnose_atom270_samecard_retry1.py "$arm"
+done
+```
+
+Each arm must use a fresh Python process within the same allocated Pod. Re-loading engines in one Python process failed with SDK INIT_INTERNAL after the first arm; that failed attempt is preserved in the research record. All output files must be absent before starting. Scripts reject overwrites. Keep the allocation unchanged and retain all four ATOM270_BINDING records.
+
+## Next investigation and acceptance
+
+- Obtain an SDK-supported intermediate-output/profiling path to identify the first divergent decode operator, preserving identical KV and forced history.
+- Evaluate any compiler/runtime candidate against both2048/index13 and512/index50 reproductions, all active lanes and KV placements, then free generation and held-out prompts.
+- Re-run matched native/whole/stream serving and fault checks before a B4 promotion. Do not change greedy tie-breaking merely to hide differing logits.
+
+Same-card raw arrays are archived outside Git; compact audit and full hashes are included. Remote synchronization is unverified. B1 remains the existing default and B4 parity remains failed; no production engine change is included.

experiments/evidence270/allcore-abba-evidence-manifest.json

@@ -0,0 +1,46 @@
+{
+  "research_event_id": "f2ce561ebec14afba60d52baad18094e",
+  "archive": "Local raw evidence; result JSONs registered with research-hub; remote synchronization unverified",
+  "files": {
+    "atom270-qwen4b-allcore-abba-a1.json": {
+      "sha256": "0a5b574e4382e9a2402b6f9f0799f0627310481fc8de9389a7379f60f200e381",
+      "bytes": 251121
+    },
+    "atom270-qwen4b-allcore-abba-a1.server0.log": {
+      "sha256": "9b7366cd6fe02e8b1d0ab75690d46c287dc13590051ed0020a3d13d1b4418bd1",
+      "bytes": 233
+    },
+    "atom270-qwen4b-allcore-abba-b1.json": {
+      "sha256": "26bfb41d507f171cd073078166b3afb346ae935451c6ce16f3a4f4e6af9594c5",
+      "bytes": 252978
+    },
+    "atom270-qwen4b-allcore-abba-b1.server0.log": {
+      "sha256": "9b7366cd6fe02e8b1d0ab75690d46c287dc13590051ed0020a3d13d1b4418bd1",
+      "bytes": 233
+    },
+    "atom270-qwen4b-allcore-abba-b2.json": {
+      "sha256": "1750d0b6a8844b8968900528f01be11ffba0da36eb7df77bc8a66c5e9e692a07",
+      "bytes": 252987
+    },
+    "atom270-qwen4b-allcore-abba-b2.server0.log": {
+      "sha256": "9b7366cd6fe02e8b1d0ab75690d46c287dc13590051ed0020a3d13d1b4418bd1",
+      "bytes": 233
+    },
+    "atom270-qwen4b-allcore-abba-a2.json": {
+      "sha256": "a7729ef1958aadcd6b70662c15f116275bebaf570a5b55df81e5be9db58e3fbe",
+      "bytes": 251128
+    },
+    "atom270-qwen4b-allcore-abba-a2.server0.log": {
+      "sha256": "9b7366cd6fe02e8b1d0ab75690d46c287dc13590051ed0020a3d13d1b4418bd1",
+      "bytes": 233
+    },
+    "atom270-allcore-abba-outer.json": {
+      "sha256": "0b5e495fbac1334417819c223ba95372717e00a621744a4e55e2277083e2eb56",
+      "bytes": 2079
+    },
+    "atom270-allcore-abba-arms.json": {
+      "sha256": "0d923ac06cbc430b304b9bf3d103b67ccb08d45261e57c85f6deb3233102b9fd",
+      "bytes": 3001
+    }
+  }
+}

experiments/evidence270/allcore-evidence-manifest.json

@@ -0,0 +1,18 @@
+{
+  "research_event_id": "5314580e2c8845538448674bd734003d",
+  "archive": "Local research-hub artifacts; remote synchronization unverified",
+  "files": {
+    "atom270-qwen4b-allcoreprobe-sdk.json": {
+      "bytes": 124334,
+      "sha256": "32f89a5887a7bf8b69561ba78c2245b2d91baf04585bc283b39f9dfae5ef555c"
+    },
+    "atom270-qwen4b-allcore-http.json": {
+      "bytes": 510017,
+      "sha256": "c33843569e3e172f9088fca2d15d4ecb9a2758788e5d345cec704a5f998ffaae"
+    },
+    "atom270-qwen4b-allcore-http.server0.log": {
+      "bytes": 235,
+      "sha256": "8557841300ed96466b41e5972a9ac5da6f5d25c426c946100a456eed77494bd8"
+    }
+  }
+}

experiments/evidence270/allcore-pd-evidence-manifest.json

@@ -0,0 +1,30 @@
+{
+  "research_event_id": "2bcac27c59d1486a8e7df59049a391c5",
+  "archive": "Local raw evidence; result JSONs registered with research-hub; remote sync unverified",
+  "files": {
+    "atom270-qwen4b-allcore-pd-whole.json": {
+      "sha256": "fa6d2950f9f6c95e74ac2525e91d8f15fe09560d4d69594cd81cae31eefb4e24",
+      "bytes": 512556
+    },
+    "atom270-qwen4b-allcore-pd-whole.server0.log": {
+      "sha256": "4e04e39584bf660cbeadbd22946f775b0c8ca6e7913af153d06257b38de4a077",
+      "bytes": 272
+    },
+    "atom270-qwen4b-allcore-pd-stream.json": {
+      "sha256": "07ca97efe2ed748d927174ea352e990988cdd8ccd0095aae160412ed90d1ab5d",
+      "bytes": 512604
+    },
+    "atom270-qwen4b-allcore-pd-stream.server0.log": {
+      "sha256": "c910abaffd603607d01ce0eae13cf96851f56cbab1351a7399087afb77c63114",
+      "bytes": 272
+    },
+    "atom270-allcore-pd-outer.json": {
+      "sha256": "703597217d53f0d358accfa77a3d52c19e29d59fa0549e5353be0ce33029010a",
+      "bytes": 2004
+    },
+    "atom270-allcore-pd-arms.json": {
+      "sha256": "69d6df206fe58d4b5135df98e40053b30063d515ff333c69ea8c4daa5d97a94f",
+      "bytes": 1709
+    }
+  }
+}

experiments/evidence270/allcore-pd-quality-evidence-manifest.json

@@ -0,0 +1,34 @@
+{
+  "research_event_id": "658d1ea450bc436cb73ebc1a484a05bc",
+  "archive": "Local raw evidence; JSONL results registered with research-hub; remote sync unverified",
+  "files": {
+    "atom270-allcore-pd-quality-outer.json": {
+      "sha256": "babe893e654293a781155dc7cc320fd004d9efa5762949ad06d45af768c2daa5",
+      "bytes": 1983
+    },
+    "atom270-qwen4b-allcore-pd-quality-whole.jsonl": {
+      "sha256": "f868c1bf6d024a8284e362ea2a89b55ba4b71cf02513023d1ba55fd581565070",
+      "bytes": 30294
+    },
+    "atom270-qwen4b-allcore-pd-quality-whole.server0.log": {
+      "sha256": "db709634d7c48970710d625606d84e52386314f3adc8dbdf58c4ccad78826be4",
+      "bytes": 301
+    },
+    "atom270-qwen4b-allcore-pd-quality-whole-run.json": {
+      "sha256": "5ee76d392020991ca918337f76d5428dd88e323c778f45b16d23e877a3c2701c",
+      "bytes": 1372
+    },
+    "atom270-qwen4b-allcore-pd-quality-stream.jsonl": {
+      "sha256": "4016161cc6d72357ae8a76b911ccc39199d53093c9f63d46fe5323cb90be6c2b",
+      "bytes": 30305
+    },
+    "atom270-qwen4b-allcore-pd-quality-stream.server0.log": {
+      "sha256": "e97e8808d5261893538e315840ccbd20aeb061ea073dc6504c0d237df7467df1",
+      "bytes": 301
+    },
+    "atom270-qwen4b-allcore-pd-quality-stream-run.json": {
+      "sha256": "791ec7f4c26fe7f4024758d48bc33c3e6b21ba50d9d927ad905eba5eee79bc95",
+      "bytes": 1395
+    }
+  }
+}

experiments/evidence270/allcore-quality-evidence-manifest.json

@@ -0,0 +1,16 @@
+{
+  "research_events": [
+    "5ceb6f2bb0644a4da66914ce79bb1fc1"
+  ],
+  "archive": "Local research-hub artifacts; remote synchronization unverified",
+  "files": {
+    "atom270-qwen4b-allcore-quality-retry1.jsonl": {
+      "sha256": "d55ad59583febee514dfc507f932d579cf32d1435eb5fcff63eb89f4001c8626",
+      "bytes": 27463
+    },
+    "atom270-qwen4b-allcore-quality-c12.jsonl": {
+      "sha256": "97dc5c9bb163f455dc6a0a85733c84de6f45dbd23f79fff0535d02a4ffa2d7be",
+      "bytes": 30295
+    }
+  }
+}

experiments/evidence270/analyze_timeline270.py

@@ -0,0 +1,47 @@
+"""Summarize host observations without inferring device utilization or DMA overlap."""
+import argparse
+from collections import Counter
+import hashlib
+import json
+from pathlib import Path
+import statistics
+
+
+def distribution(values):
+    return dict(count=len(values),sum=sum(values),min=min(values) if values else None,
+                median=statistics.median(values) if values else None,max=max(values) if values else None)
+
+
+def summarize(trace):
+    remote,decode=trace['remote'],trace['decode']
+    for row in remote+decode:
+        if row['end']<row['start']:
+            raise ValueError('negative host interval')
+    completed=sorted((r for r in remote if not r['error']),key=lambda r:r['end'])
+    ends=[r['end'] for r in completed]
+    backlogged_gaps=[b['end']-a['end'] for a,b in zip(completed,completed[1:]) if b['start']<=a['end']]
+    lane_hist=Counter(str(r['lanes']) for r in decode if not r['error'])
+    return dict(remote=dict(completed=len(remote),errors=sum(r['error'] for r in remote),
+                submit_to_completion_s=distribution([r['end']-r['start'] for r in remote]),
+                completion_gap_s=distribution([b-a for a,b in zip(ends,ends[1:])]),
+                backlogged_completion_gap_s=distribution(backlogged_gaps)),
+                decode=dict(calls=len(decode),errors=sum(r['error'] for r in decode),
+                lane_histogram=dict(sorted(lane_hist.items())),max_lanes=max((int(k) for k in lane_hist),default=0),
+                calls_with_ready_prefill=sum(r['ready']>0 for r in decode),
+                calls_with_waiting=sum(r['waiting']>0 for r in decode),
+                calls_with_prefilling=sum(r['prefilling']>0 for r in decode),
+                host_call_s=distribution([r['end']-r['start'] for r in decode])),
+                limits='All harness phases included. Completion gaps include client/phase idle periods; submit-to-completion includes producer queueing. Queue states are snapshots at decode entry, not continuous ready-time measurements. Python intervals do not prove DMA overlap; overlapping sums are not wall-clock decomposition.')
+
+
+if __name__=='__main__':
+    parser=argparse.ArgumentParser(__doc__)
+    parser.add_argument('server_log',type=Path)
+    parser.add_argument('--output',type=Path,required=True)
+    args=parser.parse_args()
+    raw=args.server_log.read_bytes()
+    rows=[json.loads(l.split(' ',1)[1]) for l in raw.decode().splitlines() if l.startswith('ATOM270_HOST_TIMELINE ')]
+    if len(rows)!=1:raise ValueError('Expected one completed host timeline')
+    out=summarize(rows[0]);out['source_sha256']=hashlib.sha256(raw).hexdigest()
+    with args.output.open('x') as f:json.dump(out,f,indent=2);f.write('\n')
+    print(json.dumps(out,indent=2))