[antonwolfy]

Summary

_is_degenerate compares the sd with 1e-9 * max(1.0, abs(mean)), so a sample whose sd is under 1e-9 counted as constant whatever its scale (#640). On main:

Input	Result on main	Now
mean_ci([1e-300, 2e-300, 3e-300], ndigits=None)	method: degenerate, lo == hi == mean	student_t, lo < mean < hi
mean_ci([1e-12, 2e-12, 3e-12], ndigits=None)	degenerate	student_t
_cohens_d(2e-300, 1e-300)	None (unknown)	2.0
paired_test of deltas 1e-300 ... 4e-300	cohens_d: None	cohens_d: 1.936

Option taken: scale before the check, as point_biserial does since #601, rather than a relative-only _is_degenerate. The floor of 1.0 keeps its job of absorbing float noise on a constant sample (a constant shift has sd ~ 3e-15), now relative to the data; dropping it would read a mean near 0 with noise-sized spread as varying.

The same cause in the sign-flip tie tolerance (EPS = 1e-9, absolute) is tracked in #762.

Closes #640

Changes

• dokimi/dokimi/eval/stats.py: new helper _is_degenerate_at_scale(mean, sd, top) divides the mean and sd by the power of two at or below top (exact) and calls _is_degenerate; top == 0 (a sample of zeros) is degenerate as before. mean_ci passes the largest absolute value, _cohens_d (signature unchanged) passes max(abs(d_mean), d_sd). The returned mean and bounds are unchanged. _is_degenerate and ZERO_VARIANCE_REL_TOL are untouched.
• dokimi/tests/test_stats_tiny_scale.py: tiny-scale intervals at 1e-300 and 1e-12, scale invariance by a power of two, constant and float-noise samples that stay degenerate, _cohens_d and paired_test on tiny deltas.
• dokimi/docs/benchmarking-and-comparison.md: says "constant" for d_z is judged at the deltas' own scale.
• Changelog fragment (Fixed).

Risk

A sample whose values differ by less than 1e-9 in absolute terms but are not constant at their own scale now gets a Student-t interval (method: student_t) from mean_ci and a Cohen's d from paired_test and the other users of _describe_deltas (arms, interactions), where it got degenerate / unknown. A sample that is constant at its own scale (exact repeats, or float-noise spread of 1e-9 relative) stays degenerate. Deltas that are all float noise around 0, such as [5.5e-17, 0.0, 5.5e-17, 0.0, 0.0] (0.3 - 0.2 - 0.1), have a mean and sd of the same tiny size, so they now get a d_z (-0.73 for that vector) instead of unknown; scaling from the deltas instead of the mean and sd gives the same, since noise is the whole scale there. Samples at the scale of 1 or above are decided as before, including the existing big / small pair and the constant-shift test.

Testing

On Linux, at 65008c5 (on main 7137604):

• make ci-local: 33 passed, 0 failed, 14 skipped (docker external-catalog smoke, Claude Code fidelity install, matrix legs and release jobs that do not apply on a PR, micromamba, semgrep, fuzz, cli-interface package). 20449 tests passed, 19 skipped; changed-line coverage 100%.
• The repro above fails on main and passes here: 5 of the 8 new tests fail on main's code, and the 3 that stay degenerate pass on both.
• Mutants, one break per copy: mean_ci back to _is_degenerate(mu, sd) fails the two tiny-scale interval tests and the scale-invariance test; _cohens_d back to _is_degenerate(d_mean, d_sd) fails test_cohens_d_of_tiny_deltas_is_a_number and the paired_test test; the helper returning sd == 0 (no noise floor) fails the float-noise test and the constant-shift _cohens_d test; the helper returning False fails those two and the constant-tiny-sample test.
• Not run: the matrix legs on Python 3.11, 3.12 and 3.14.

Revert

A plain revert restores the absolute floor. Nothing is stored.

Checklist

• Tests added/updated and pytest passes
• ruff, mypy, and schema checks pass
• Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
• Self-reviewed per "Before opening a PR" in AGENTS.md
• No ad-hoc run artifacts committed
• New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

[exai-bot]

✅ OpenCodeReview: Review complete: 0 finding(s) across 4 selected item(s).

CI at d8b869d: 25 passing, 3 skipped.

Review details

• Commit reviewed: d8b869d against merge base 7137604
• Coverage: 4 files in 1 groups
• Engine: OpenCodeReview v1.13.5-exai.4.8f73eada, us.openai.gpt-6.1-sol
• Usage: 122k tokens (81k from cache), 24s
• Tool calls: 6 (4 file_read, 1 code_search, 1 pr_comments)
• Tiny-scale degenerate statistics handling, tests, and documentation: 4 files

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.72%. Comparing base (7137604) to head (d8b869d).
⚠️ Report is 1 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #763   +/-   ##
=======================================
  Coverage   95.72%   95.72%           
=======================================
  Files         415      415           
  Lines       55363    55365    +2     
  Branches     9516     9516           
=======================================
+ Hits        52994    52996    +2     
  Misses       1466     1466           
  Partials      903      903           

Files with missing lines	Coverage Δ
dokimi/dokimi/eval/stats.py	97.82% <100.00%> (+0.02%)	⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[vchamarthi]

Pushed 1 commit to fix/degenerate-tiny-scale (65008c5..d8b869d).

• Removed the top == 0 branch from _is_degenerate_at_scale; zeros still read as degenerate through _is_degenerate.
  Run: the full suite with main merged in, 20494 passed.

The PR description needs these corrections:

• "top == 0 (a sample of zeros) is degenerate as before." should read: A sample of zeros is degenerate as before.