position-bias

[NeurIPS'2026] Position Bias in Mamba and Hybrid Language Models | Evidence Position Bias Across Sequence Mixers in Long-Context Question Answering

ACL 2026 | CapCal: content-agnostic probability calibration for de-biasing listwise rerankers.

Option-reversal control for paired binary vision–language benchmarks: estimates a model's answer-slot bias and perceptual accuracy from one extra inference pass per question, attributes each failure to slot or perception, and reports scores against the correct chance floors. Ongoing project; code only.

SSIT: a label-free, gold-free test for whether LLM-judge position x verbosity bias corrections actually compose. No human labels, no model of the judge. Code, 7-judge/6-family pilot data, and pre-registered protocol for the NeurIPS 2026 JUDGe workshop paper.

Learning-to-rank lab: listwise models, position bias, calibration, and NDCG evaluation

Position-bias-aware ranking: estimating and correcting position bias to optimize Earnings Per Visitor (EPV) · simulation study · IPW & propensity modeling · Python

position bias in LLM judges is worse than people think: ask a small instruct model to choose 1 of 2 items. 81% of the answer will be based on which slot the item was in, not which it was. swap the order and the answer flips 78% of the time. an inconclusive cognitive dissonance experiment

Blind side-by-side human eval of two model responses on a weighted rubric — randomized panes to kill position bias, bootstrap CIs on the margin, inter-rater reliability. An eval you can't audit is a vote, not a measurement.

Counterfactual learning-to-rank for marketplace search logs in PySpark: position-bias estimation, IPS-weighted training, NDCG evaluation against known ground truth

Open-source implementation of the typed-decision pattern popularised by TypeSafe's Jev: read a decision out of a small language model's logits, in the browser. Library, benchmarks and paper. Not affiliated with TypeSafe.

Measure position/verbosity/assertiveness bias in an LLM-as-judge (Claude) by judging pairs in both orders. Finding: no position bias, but 75% verbosity bias and 100% assertiveness bias — judge scores gameable by length + tone.

Two kinds of saturation: why LLM-judge order bias is hard to measure — essay, Lean proofs (0 sorry), and a reproducible dispersion measurement.

Ranking evaluation with error bars: NDCG, MRR and MAP with confidence intervals, plus position-bias correction for click logs. No dependencies.

Monthly bias audits of LLM judges

Official reproduction code for 'Efficient LLM Preference Classification Through Position Bias Mitigation and Architectural Symmetry' (Applied Intelligence APIN-D-26-05359)