[WS2][Mismatch] Add Qwen3 FFN implementation factor

.github/workflows/ci.yml

@@ -76,6 +76,10 @@ jobs:
         run: |
           python -m pytest tests/test_kv_cache_attention.py -v -k "not large and not gpu"
 
+      - name: Run Mismatch Framework Tests (CPU-safe)
+        run: |
+          python -m pytest tests/test_mismatch_framework.py tests/test_mismatch_logprob_adapter.py -v
+
   docs:
     runs-on: ubuntu-latest
     steps:

rl_engine/mismatch/README.md

@@ -0,0 +1,122 @@
+<!-- SPDX-License-Identifier: Apache-2.0 -->
+<!-- Copyright (c) 2026 RL-Kernel Contributors -->
+
+# `rl_engine/mismatch` — detecting and reporting training-inference mismatch
+
+The rollout policy `π_old` and the training policy `π_θ` compute logprobs for the
+**same tokens with the same weights** and still disagree. This package turns
+"which of the dozens of possible causes is it" into named **factors**, each a
+switch that can be flipped one at a time and attributed to a side.
+
+The output is a report: which factors were measured, which could not be and why,
+and the few most suspicious modules ranked.
+
+## Why the tail, not the mean
+
+`dlogp = log π_θ − log π_old` enters the GRPO objective through `ρ = exp(dlogp)`,
+which the objective clips at `1 ± ε`. With `ε = 0.2`, any token past
+`|dlogp| > ln(1.2) ≈ 0.182` has its **gradient signal discarded** — and not
+random tokens, the most mismatched ones.
+
+A healthy `dlogp_mean` is `0.002–0.008` (dense) or `0.01–0.03` (large MoE), all
+far below that edge. **Judging on the mean alone always concludes "everything is
+fine."** Hence `dlogp_p99` / `dlogp_max` / `clip_fraction` / `worst_token`, and a
+diagnosis matrix that converges on `clip_fraction`.
+
+## Four arms, then four gates
+
+A factor expands into four arms, not on/off — only a one-sided swap identifies a
+side, and only a two-sided swap proves the reference itself is sound:
+
+| arm | rollout | training | what it buys |
+|---|---|---|---|
+| `both_native` | native | native | the baseline the others are measured against |
+| `both_reference` | reference | reference | **self-check gate** — must be bitwise identical, or this factor's conclusions are void |
+| `training_reference_only` | native | reference | deviation gone ⇒ training side is the source |
+| `rollout_reference_only` | reference | native | deviation gone ⇒ rollout side is the source |
+
+A factor with no reference implementation is a **parameter sweep** instead: one
+arm per allowed value. A sweep measures; it cannot conclude, because nothing was
+swapped and so there is no side to attribute to.
+
+Before any verdict, four gates run. **"Not measured" and "measured and clean"
+are different things**, and confusing them is the mistake an attribution
+framework is most likely to make:
+
+| gate | fails when | verdict |
+|---|---|---|
+| 1 · did it apply | any arm is not `APPLIED` | `VARIANT_DID_NOT_APPLY`, with the resolution trace |
+| 2 · evidence | `required_evidence` incomplete | `INSUFFICIENT_EVIDENCE` |
+| 3 · shards | fewer logprob shards than `world_size` | `INSUFFICIENT_EVIDENCE` |
+| 4 · guards | a pitfall guard failed | `INSUFFICIENT_EVIDENCE` |
+
+`SwitchStatus.FELL_BACK` is why gate 1 exists: the reference was requested, the
+engine silently reverted to native, and "the deviation did not change" then reads
+as a clean `NOT_THIS_FACTOR`.
+
+## Noise floors
+
+A result only means something at a floor that can resolve it. Each step down
+adds exactly one new noise source, so a failure points at a known suspect set. A
+floor that has not passed blocks the next.
+
+| floor | configuration | new noise source |
+|---|---|---|
+| `SINGLE_LAYER_ANCHOR` | 1 layer, single device, determinism on | none — failing bitwise here is an **operator bug**, not mismatch |
+| `FULL_MODEL_SINGLE_GPU` | all layers, single device | accumulation over depth |
+| `SHARDED_SINGLE_NODE` | TP + SP on one node | reduction order — the first floor with *real* mismatch |
+| `PRODUCTION` | target TP/CP/PP, determinism off, decode | everything else; the only floor readable against `EXPECTED_RANGES` |
+
+## Layout
+
+```
+mismatch/
+├── schema/              pure data types, frozen, no behaviour
+├── pipeline/            registry → planner → runner → diagnosis → report
+├── engines/             the two sides under test: megatron.py, vllm.py
+├── reference_adapters/  delivering pinned settings, and reading them back
+├── model_meta/          per-model correspondence and call chain (qwen3.py)
+├── operator_checks/     plugins, one directory per operator
+├── docs/                tutorials
+└── __main__.py          CLI, and the only module that imports plugins
+```
+
+`engines/` holds **`megatron.py` and `vllm.py` and nothing else** — the two
+policies as they really run, shared across operators. Anything that merely
+satisfies `ScoringBackend` is a harness, not a side under test, and lives in
+`tests/` (see `tests/mismatch_cpu_backend.py`). The line is role, not protocol.
+
+Three dependency rules keep the plugin seam open:
+
+1. `schema/` never imports `pipeline/`; inside `schema/`, `values.py` imports
+   nothing from the project.
+2. `pipeline/` never imports `operator_checks/` — it sees only what the registry
+   hands it. Break this and adding an operator becomes changing the framework.
+3. Only `__main__` imports `operator_checks/`, which triggers registration.
+
+## Running it
+
+```bash
+python -m rl_engine.mismatch list                    # operators and their factors
+python -m rl_engine.mismatch plan --gpu-count 2      # expand into cases, cheapest first
+python -m rl_engine.mismatch plan --json
+```
+
+The attention, GEMM and logprob adapters are wired end to end. Attention fails
+closed when an engine does not report the actual Split-KV plan set, CP block
+manifest, collective trace, or RoPE evidence. GEMM covers the Qwen3 FFN
+implementation and RowParallel forward reduction factors. Logprob remains the
+smaller worked example of the adapter layer.
+
+## Adding to this package
+
+**Adding an operator is adding a directory; adding a factor is adding a file.**
+No existing file changes, apart from one line in `__main__._OPERATOR_PACKAGES`
+for a new operator. If your change needs an edit inside `pipeline/`, a global
+dict, or another operator's directory, the framework is missing an abstraction —
+raise it rather than patching around it.
+
+| you want to | read |
+|---|---|
+| add a kernel's factor | [`docs/add-a-kernel-factor.md`](docs/add-a-kernel-factor.md) |
+| add a communication feature | [`docs/add-a-comm-feature.md`](docs/add-a-comm-feature.md) |

rl_engine/mismatch/__init__.py

@@ -0,0 +1,27 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (c) 2026 RL-Kernel Contributors
+
+"""Training-inference mismatch diagnosis, one factor at a time.
+
+Rollout and training compute logprobs for the same tokens with the same weights
+and still disagree. This turns "which of the dozens of possible causes is it"
+into switches that can be flipped one at a time and attributed to a side.
+
+Three dependency rules keep the plugin seam open:
+
+1. ``schema/`` never imports ``pipeline/``, and inside ``schema/`` the imports go
+   one way, with ``values.py`` importing nothing from the project.
+2. ``pipeline/`` never imports ``operator_checks/``; it sees only what the
+   registry hands it. Break this and adding an operator becomes changing the
+   framework.
+3. Only ``__main__`` imports ``operator_checks/``, to trigger self-registration.
+
+See ``README.md`` for the layout and ``docs/`` for how to add a factor.
+"""
+
+from rl_engine.mismatch import pipeline, schema
+
+__all__ = [
+    "pipeline",
+    "schema",
+]

rl_engine/mismatch/__main__.py

@@ -0,0 +1,152 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (c) 2026 RL-Kernel Contributors
+
+"""Command line entry point, and the only place that imports operator plugins.
+
+Importing them here is what triggers self-registration, and it keeps the
+dependency arrow pointing one way: ``pipeline/`` never reaches into
+``operator_checks/``.
+"""
+
+from __future__ import annotations
+
+import argparse
+import importlib
+import json
+import sys
+from typing import Sequence
+
+from rl_engine.mismatch.pipeline import (
+    OPERATOR_CHECKS,
+    build_variants,
+    missing_prerequisites,
+    order_cases_by_rebind_cost,
+    reject_contradictory_factors,
+)
+from rl_engine.mismatch.schema import NoiseFloor
+
+# An operator that is not listed here does not exist as far as the framework
+# is concerned.
+_OPERATOR_PACKAGES: tuple[str, ...] = (
+    "rl_engine.mismatch.operator_checks.gemm",
+    "rl_engine.mismatch.operator_checks.attention",
+    "rl_engine.mismatch.operator_checks.logprob",
+)
+
+
+def load_operator_plugins(packages: Sequence[str] = _OPERATOR_PACKAGES) -> tuple[str, ...]:
+    """Import each plugin package so its decorator runs."""
+
+    for package in packages:
+        importlib.import_module(package)
+    return OPERATOR_CHECKS.operators()
+
+
+def build_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        prog="python -m rl_engine.mismatch",
+        description="Diagnose training-inference mismatch, one factor at a time.",
+    )
+    sub = parser.add_subparsers(dest="command", required=True)
+
+    listing = sub.add_parser("list", help="list registered operators and their factors")
+    listing.add_argument("--operator", default=None)
+
+    plan = sub.add_parser("plan", help="expand factors into variants without running anything")
+    plan.add_argument("--operator", default=None)
+    plan.add_argument(
+        "--noise-floor",
+        default=NoiseFloor.SINGLE_LAYER_ANCHOR.value,
+        choices=[floor.value for floor in NoiseFloor],
+    )
+    plan.add_argument("--gpu-count", type=int, default=0)
+    plan.add_argument("--json", action="store_true")
+
+    return parser
+
+
+def command_list(operator: str | None) -> int:
+    operators = load_operator_plugins()
+    if not operators:
+        print(
+            "no operator plugins registered.\n"
+            "The framework ships without operators: add one under "
+            "rl_engine/mismatch/operator_checks/<operator>/ and list it in "
+            "__main__._OPERATOR_PACKAGES.\n"
+            "See rl_engine/mismatch/operator_checks/__init__.py for the layout."
+        )
+        return 0
+
+    for name in operators:
+        if operator is not None and name != operator:
+            continue
+        factors = OPERATOR_CHECKS.factors_for(name)
+        print(f"{name}: {len(factors)} factors")
+        for factor in factors:
+            print(f"  {factor.id:<40} {factor.category.value}")
+    return 0
+
+
+def command_plan(operator: str | None, noise_floor: str, gpu_count: int, as_json: bool) -> int:
+    load_operator_plugins()
+    factors = OPERATOR_CHECKS.factors_for(operator)
+    if not factors:
+        print("nothing to plan: no operator plugins are registered.")
+        return 0
+
+    reject_contradictory_factors(factors)
+
+    runnable = []
+    skipped = []
+    for factor in factors:
+        unmet = missing_prerequisites(factor, gpu_count=gpu_count)
+        if unmet:
+            skipped.append((factor, unmet))
+        else:
+            runnable.append(factor)
+
+    cases = [(factor, variant) for factor in runnable for variant in build_variants(factor)]
+    ordered = order_cases_by_rebind_cost(cases)
+
+    if as_json:
+        payload = {
+            "noise_floor": noise_floor,
+            "runnable_factors": [factor.id for factor in runnable],
+            "skipped": {factor.id: [item.reason for item in unmet] for factor, unmet in skipped},
+            "cases": [
+                {
+                    "factor": factor.id,
+                    "variant": variant.name,
+                    "rebind_cost": factor.switch.rebind_cost.value,
+                    "switch_values": dict(variant.switch_values),
+                }
+                for factor, variant in ordered
+            ],
+        }
+        print(json.dumps(payload, indent=2, sort_keys=True))
+        return 0
+
+    print(f"noise floor: {noise_floor}")
+    print(f"runnable factors: {len(runnable)}   cases: {len(ordered)}")
+    if skipped:
+        print("\nskipped (prerequisites not met):")
+        for factor, unmet in skipped:
+            for item in unmet:
+                print(f"  {factor.id}: {item.reason}")
+    print("\ncases in execution order (cheapest rebuild first):")
+    for factor, variant in ordered:
+        print(f"  [{factor.switch.rebind_cost.value:<22}] {factor.id} :: {variant.name}")
+    return 0
+
+
+def main(argv: Sequence[str] | None = None) -> int:
+    args = build_parser().parse_args(argv)
+    if args.command == "list":
+        return command_list(args.operator)
+    if args.command == "plan":
+        return command_plan(args.operator, args.noise_floor, args.gpu_count, args.json)
+    return 1
+
+
+if __name__ == "__main__":  # pragma: no cover
+    sys.exit(main())

rl_engine/mismatch/docs/README.md

@@ -0,0 +1,11 @@
+<!-- SPDX-License-Identifier: Apache-2.0 -->
+<!-- Copyright (c) 2026 RL-Kernel Contributors -->
+
+# Tutorials
+
+Concepts live in [`../README.md`](../README.md); these are the how-to.
+
+| tutorial | when |
+|---|---|
+| [add-a-kernel-factor.md](add-a-kernel-factor.md) | a kernel computes something different on the two sides |
+| [add-a-comm-feature.md](add-a-comm-feature.md) | the suspect is a collective: a reduction order, a rewrite, a CP merge, a backend |