sign-flip tie tolerance EPS = 1e-9 is absolute: deltas that vary at a tiny scale are ties and get no p #762

dokimi/dokimi/eval/sign_flip.py:55 sets EPS = 1e-9 and treats every delta with abs(x) <= EPS as a tie (sign_flip.py:75, :126, :205; stats.py:613 n_changed). The same constant is imported by repos/compare.py (:498-:604, ties, up/down counts) and a copy _EPS = 1e-9 is in eval/paired_cluster.py:47.

The cause is the same as #640 (an absolute epsilon read as a property of any scale), and it is the part #640's fix does not cover.

Repro:

from dokimi.eval import stats
r = stats.paired_test([2e-300, 4e-300, 3e-300, 5e-300, 6e-300, 9e-300, 7e-300, 8e-300], [1e-300] * 8)
# p None, n_changed 0, significant None, cohens_d 1.837, effect 'large'
r = stats.paired_test([2e-3, 4e-3, 3e-3, 5e-3, 6e-3, 9e-3, 7e-3, 8e-3], [1e-3] * 8)
# p 0.0078, n_changed 8, significant True, cohens_d 1.837

So at 1e-300 the row has an effect size of 'large' and no p, because every pair is a tie. The tolerance exists for float noise on scores (a delta of 1e-13 on a score near 1), so a relative tie rule (scale EPS by the larger of the pair's magnitudes, or by the delta vector's power-of-two scale) is the fix to decide; callers whose deltas are scores in [0, 1] may prefer to keep the absolute tolerance, which is the decision for the maintainers.

Found while fixing #640. The fix's PR is #763.