[llying-001]

Description

Stacked on #210 (base feat/sglang-mix-glm5.2-tp4-v0522); retarget to main once #210 merges. Only the last two commits are this PR.

Under DP attention every decode step waits for the busiest of the decode ranks, and that rank's step time grows with its request count as well as its context. --decode-input-demand on balances in-flight input tokens only, so a rank can hold many short requests for another's one long request. In #210's sweep this shows up as interactivity that barely moves with concurrency (2P1D p90 35.7 at C176, 40.8 at C96).

This adds --decode-request-weight W (INFERA_DECODE_REQUEST_WEIGHT, default 0 = unchanged): with decode input demand on (or shadow), each Decode candidate costs

in-flight input tokens + this request's tokens + W × requests the target holds

Held requests are the existing demand reservations (booked at the pick, released at response EOF, cancellation or failure), so this is the same count Dynamo charges with router-decode-active-request-weight (decode blocks + weight × active requests; GB300's GLM-5.2 recipe uses 8192 blocks per request, i.e. request counts dominate).

Type of change

• Documentation change (change only to the documentation, either a fix or a new content)
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Infra/Build change
• Code refactoring

Changes

• rust/router/src/config.rs: --decode-request-weight <tokens> (env INFERA_DECODE_REQUEST_WEIGHT, default 0); rejected unless --decode-input-demand is shadow or on.
• rust/router/src/policy.rs: the demand cost adds weight × held requests from the ledger; the decode_demand debug event carries the weight. In shadow the suggestion uses the weighted cost and selection stays legacy.
• rust/router/src/metrics.rs, decode_demand.rs: infera_router_decode_demand_requests{target} gauge, set on book and release.
• infera/server/args.py, launch_rust.py: the same flag on the Python launcher, forwarded to the Rust router only when given (Rust-only, like --decode-input-demand).
• Tests: decode_request_weight_balances_requests_held, decode_request_weight_in_shadow_keeps_the_legacy_pick, decode_request_weight_requires_decode_demand; launcher forwarding / validation in tests/unit/server/test_launch_rust.py.
• Docs: manual/reference/cli.md, manual/features/kv_aware_routing.md (Request-count weight), examples/sglang_2p1d_glm5.2/README.md (Decode request balancing, with the data below).

Validation

Unit (in the lmsysorg/sglang-rocm:v0.5.22-rocm720-mi35x-20261018 base, cargo 1.99.1): cargo test --release -p infera-router 354 + 46 + 4 + 14 passed, 0 failed; cargo fmt --check clean; pytest tests/unit/server/test_launch_rust.py 25 passed; ruff format --check / ruff check clean on the touched Python files.

AgentX, GLM-5.2-MXFP4, 1P1D C88 with #210's kit (PREFILL_WORKERS=1, prefill on GPUs 4-7, forced acceptance 3.61) on two AUS MI355X nodes (smci355-ccs-aus-n04-25 prefill / n06-33 decode). The image is #210's Dockerfile.sglang; the router binary was built from this branch in the same base and mounted with INFERA_ROUTER_BIN. Prefills carry the decode's EAGLE flags (#210 note 10). Both points on one deployment, one after the other; before each the router was restarted with its weight and the prefill's cache flushed (/flush_cache, checked: every prefill rank's first batch had 0 cached tokens). 10 warm-up requests per lane, 1800 s measured (the job was preempted every 0.5-1 h, so a 3600 s point rarely completed):

--decode-request-weight	tok/s/GPU	requests (per s)	interactivity p50 / p90	TTFT p50 / p90 / mean	ITL p50 / p90
0 (= #210)	26,462	5,160 (2.87)	75.8 / 55.7	5.46 / 11.09 / 8.29 s	13.2 / 18.0 ms
1,000,000	27,186	5,184 (2.88)	86.5 / 79.6	5.61 / 12.01 / 9.50 s	11.6 / 12.6 ms

Decode ranks over the measured window (decode log, #running-req / #token per rank, 3 s bins):

weight	running per rank	busiest median / p90	requests busiest / mean (median / p90)	context busiest / mean (median / p90)	step median
0	4.66	8 / 11	1.67 / 2.13	1.23 / 1.44	44.4 ms
1,000,000	3.95	6 / 8	1.50 / 1.81	1.52 / 1.84	41.2 ms

• p90 interactivity +43% at the same throughput. At equal busiest-rank load the two points take the same time per step (6 requests: 40.5 / 40.8 ms, 8: 43.6 / 43.5 ms, 9: 49.5 / 47.7 ms); the difference is that the busiest rank no longer reaches 10-14 requests.
• TTFT rises (mean +15%, p90 +8%): context spreads less evenly, so more KV lands on fewer ranks. Weights between 0.18M and 1M, or a KV cap per rank, are the next things to try.
• Why a million: on this point one more request on the busiest rank cost as much step time as ~1.05M more context tokens there (least-squares fit of step time against the busiest rank's requests and tokens plus the totals, R² 0.86); on 2P1D C176 with the v0.5.21 image the same fit gives 0.17-0.19M. A 180,000 weight on another node pair (n02-29 / n04-33) gave 27,577 tok/s/GPU at 85.0 / 74.4.
• 1800 s windows understate tok/s/GPU against 3600 s by ~6% (in a 3600 s run the first half averages 113k tokens per request, the whole 130k) and keep p90 interactivity within 2%. Canonical 3600 s points are running and will be added here.

Data, scripts and analysis: llying/1p1d-decode-balance-aus-20261021/ on dev/pd_opt/glm_5.2_agentx.

Checklist:

• The functionality is complete
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[llying-001]

Folded into #210 (its three commits now sit on #210's branch, rebased onto main adbb9dc5 and signed off: 565461b3 router, 1082dc8d docs, 3ad7527c the 2P1D example balances the decode ranks by held requests by default, DECODE_REQUEST_WEIGHT=1000000). Closing this one; review there.

[llying-001]

Correction: GitHub marked this PR merged rather than closed, because its commits are now on its base branch (#210's feat/sglang-mix-glm5.2-tp4-v0522). Nothing reached main; the change is reviewed and merged as part of #210.