3 explainers repeat a false "CustodyStatus alone reconstructs the racial gap" narrative with fabricated feature-importance numbers

[yakew7]

Where: explainers/how-ai-detects-patterns.md (lines ~27, ~34-38, ~55), explainers/ai-objectivity-myth.md (~line 43), and explainers/ml-bias.md (~lines 75, 79) all repeat the same claim: dropping race from the COMPAS model while keeping CustodyStatus leaves the fairness gap "barely moved," because CustodyStatus allegedly has feature importance ≈0.31 (vs. race's 0.18).

The gap - fabricated importances: training the real COMPAS/unfair.py model and reading feature_importances_ directly:

import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

df = pd.read_csv('COMPAS/compas-scores-raw.csv')
df = df[df['Ethnic_Code_Text'].isin(['African-American', 'Caucasian'])]
df = df[df['DisplayText'] == 'Risk of Recidivism']
df['is_high_risk'] = df['ScoreText'].apply(lambda x: 1 if x in ['High', 'Medium'] else 0)
df['race_binary'] = df['Ethnic_Code_Text'].map({'African-American': 1, 'Caucasian': 0})
X = pd.get_dummies(df[['Sex_Code_Text', 'race_binary', 'CustodyStatus', 'MaritalStatus']])
y = df['is_high_risk']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
model = RandomForestClassifier(random_state=42)
model.fit(X_train, y_train)
imp = pd.Series(model.feature_importances_, index=X.columns)
print('race_binary:', imp['race_binary'])                                    # 0.4208
print('CustodyStatus total:', imp[[c for c in X.columns if c.startswith('CustodyStatus')]].sum())  # 0.0517

Real race_binary importance is 0.42 (more than double the claimed 0.18); real CustodyStatus total importance is 0.05 (roughly a sixth of the claimed 0.31).

The gap - "gap barely moves" is false: dropping only race (keeping CustodyStatus, Sex_Code_Text, MaritalStatus) and re-running:

X = pd.get_dummies(df[['Sex_Code_Text', 'CustodyStatus', 'MaritalStatus']])  # race dropped
# ... same train/test/fit ...
# Black: 67.88%, White: 49.50%, gap: 18.38 pp

The gap falls from 86.77pp to 18.38pp - a ~79% reduction from dropping race alone, not "barely moves." Dropping both race and CustodyStatus (matching fair.py) reaches 15.69pp - so CustodyStatus's own marginal contribution beyond race is only about 2.7pp, the opposite of "CustodyStatus alone reconstructs most of the pattern."

Also: how-ai-detects-patterns.md's own code block (line ~50, df[['Sex_Code_Text', 'MaritalStatus', 'race', 'CustodyStatus']]) and ml-bias.md's "Concrete Example" block (~lines 81-110, pd.read_csv("COMPAS/compas-scores-raw.csv") then selecting age/priors_count/juv_fel_count/juv_misd_count/sex/c_charge_degree/race/is_recid) both use the wrong ProPublica schema (same bug class as #601/#621/#622/#646) and raise KeyError when actually run.

Fix direction: correct all three files' feature-importance numbers and "barely moves" narrative to match the real, verified figures above; fix the two broken code blocks to use the real compas-scores-raw.csv schema (Ethnic_Code_Text/race_binary, ScoreText-derived target, matching COMPAS/unfair.py's recipe).