Where: faircode/report.py's compare_to_terminal() and compare_to_html() vs. assets/profiler-compare.js, which already special-cases a kind_mismatch dimension with a dedicated "comparison skipped" badge (fixed for #519).
The gap: faircode/compare.py's _compare_dimension() deliberately zeroes psi/tvd and sets drift_level="none"/groups=[] for a dimension that's banded on one side and raw on the other (e.g. age as "18-30" strings vs. raw numeric ages) - the comparison genuinely can't be measured, not because there's no drift. report.py's renderers (grep -c kind_mismatch faircode/report.py → 0) never learned to special-case this and render the zeroed placeholder as if it were a real "no drift" result, while simultaneously emitting the flag saying the comparison was skipped.
Repro (age banded as strings in one profile, left as raw numbers in the other):
$ python3 -c "... compare_to_terminal(compare(profile(df_a, {'DOB':'age'}, {}), profile(df_b, {'DOB':'age'}, {}), 'a.csv','b.csv')) ..."
DOB [age] PSI 0.000 (none drift)
score 100 → 95 (-5) TVD 0.000
...
DRIFT FLAGS
! DOB: age values are banded (e.g. "18-30") in one dataset but left raw in the other - drift comparison skipped
The same self-contradiction ("none drift" badge, PSI 0.000/TVD 0.000, empty group table) appears in compare_to_html()'s output too, alongside the "drift comparison skipped" flag text right next to it. tests/test_report.py has no test exercising a kind_mismatch dimension, so nothing currently catches this.
Fix direction: give both Python renderers the same "comparison skipped" badge/placeholder assets/profiler-compare.js already has for kind_mismatch, instead of rendering the zeroed metrics as a genuine result.
Where:
faircode/report.py'scompare_to_terminal()andcompare_to_html()vs.assets/profiler-compare.js, which already special-cases akind_mismatchdimension with a dedicated "comparison skipped" badge (fixed for #519).The gap:
faircode/compare.py's_compare_dimension()deliberately zeroespsi/tvdand setsdrift_level="none"/groups=[]for a dimension that's banded on one side and raw on the other (e.g. age as"18-30"strings vs. raw numeric ages) - the comparison genuinely can't be measured, not because there's no drift.report.py's renderers (grep -c kind_mismatch faircode/report.py→ 0) never learned to special-case this and render the zeroed placeholder as if it were a real "no drift" result, while simultaneously emitting the flag saying the comparison was skipped.Repro (age banded as strings in one profile, left as raw numbers in the other):
The same self-contradiction (
"none drift"badge,PSI 0.000/TVD 0.000, empty group table) appears incompare_to_html()'s output too, alongside the "drift comparison skipped" flag text right next to it.tests/test_report.pyhas no test exercising akind_mismatchdimension, so nothing currently catches this.Fix direction: give both Python renderers the same "comparison skipped" badge/placeholder
assets/profiler-compare.jsalready has forkind_mismatch, instead of rendering the zeroed metrics as a genuine result.