Skip to content

faircode/report.py's compare renderers never got the #519 kind_mismatch fix - self-contradictory drift reports #619

Description

@yakew7

Where: faircode/report.py's compare_to_terminal() and compare_to_html() vs. assets/profiler-compare.js, which already special-cases a kind_mismatch dimension with a dedicated "comparison skipped" badge (fixed for #519).

The gap: faircode/compare.py's _compare_dimension() deliberately zeroes psi/tvd and sets drift_level="none"/groups=[] for a dimension that's banded on one side and raw on the other (e.g. age as "18-30" strings vs. raw numeric ages) - the comparison genuinely can't be measured, not because there's no drift. report.py's renderers (grep -c kind_mismatch faircode/report.py → 0) never learned to special-case this and render the zeroed placeholder as if it were a real "no drift" result, while simultaneously emitting the flag saying the comparison was skipped.

Repro (age banded as strings in one profile, left as raw numbers in the other):

$ python3 -c "... compare_to_terminal(compare(profile(df_a, {'DOB':'age'}, {}), profile(df_b, {'DOB':'age'}, {}), 'a.csv','b.csv')) ..."
DOB  [age]    PSI 0.000  (none drift)
  score 100 → 95 (-5)    TVD 0.000
...
DRIFT FLAGS
  ! DOB: age values are banded (e.g. "18-30") in one dataset but left raw in the other - drift comparison skipped

The same self-contradiction ("none drift" badge, PSI 0.000/TVD 0.000, empty group table) appears in compare_to_html()'s output too, alongside the "drift comparison skipped" flag text right next to it. tests/test_report.py has no test exercising a kind_mismatch dimension, so nothing currently catches this.

Fix direction: give both Python renderers the same "comparison skipped" badge/placeholder assets/profiler-compare.js already has for kind_mismatch, instead of rendering the zeroed metrics as a genuine result.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinghelp wantedExtra attention is needed

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions