Where: explainers/confounding-variable.md (~lines 148-165), the "Applying it to the COMPAS audit" example:
result = check_confounding(
df=df,
feature_col="CustodyStatus",
outcome_col="prediction",
confounder_col="race",
protected_col="race", # <- same value as confounder_col
)The gap: check_confounding() computes result["confounder_vs_protected"] as crosstab(df[confounder_col], df[protected_col]) - with both parameters set to "race", this is crosstab(race, race), a trivial self-comparison, not a test of "CustodyStatus is also strongly associated with race" as the adjacent comment claims.
Repro: reconstructing the real COMPAS pipeline (per unfair.py's recipe) and running the exact call above: the real marginal chi2 (CustodyStatus vs. prediction) is ≈85.2, not the claimed 412.7; the self-compared "confounder_vs_protected" chi2 (race vs. itself) is ≈3250, not the claimed 389.2 - both numbers are wrong specifically because the code doesn't test what the surrounding prose says it tests.
Fix direction: the call should pass a genuinely different confounder from the protected attribute - e.g. confounder_col="CustodyStatus", protected_col="race" is presumably what was intended (matching the comment's own description) - and the claimed chi2/p-value output should be regenerated from an actual run with the corrected call.
Where:
explainers/confounding-variable.md(~lines 148-165), the "Applying it to the COMPAS audit" example:The gap:
check_confounding()computesresult["confounder_vs_protected"]ascrosstab(df[confounder_col], df[protected_col])- with both parameters set to"race", this iscrosstab(race, race), a trivial self-comparison, not a test of "CustodyStatus is also strongly associated with race" as the adjacent comment claims.Repro: reconstructing the real COMPAS pipeline (per
unfair.py's recipe) and running the exact call above: the real marginal chi2 (CustodyStatus vs. prediction) is ≈85.2, not the claimed 412.7; the self-compared "confounder_vs_protected" chi2 (race vs. itself) is ≈3250, not the claimed 389.2 - both numbers are wrong specifically because the code doesn't test what the surrounding prose says it tests.Fix direction: the call should pass a genuinely different confounder from the protected attribute - e.g.
confounder_col="CustodyStatus", protected_col="race"is presumably what was intended (matching the comment's own description) - and the claimed chi2/p-value output should be regenerated from an actual run with the corrected call.