Skip to content

Flaky test_a_uniform_change_shows_in_every_decile at half-penny shifts #2500

Closed

Description

@MaxGhenis

test_a_uniform_change_shows_in_every_decile (the aggregate-impact decile tests, from #2335) is flaky. It fails at seed=0, shift=0.005.

  • Why: the comparison treats a change of half a penny or less as no change, so a uniform shift that small doesn't "show in every decile", as the test expects.
  • Effect: the test can fail CI at random on any PR.
  • Fix: have the strategy exclude shifts at or below the half-penny threshold, or assert on shifts above it only. Don't change the threshold.
  • Found by: the hub's independent review of Compare two datasets on one model in the aggregate-impact tool #2425 (10/19).

Activity

  1. self-assigned this
    on Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions