Act II: The Model Blind to Harm

Step 5Live-room allocation: 3:00

The instrument is on. The view is still blocked. Illustration: Brian Stauffer.

Act I followed the number before it reached the model. This step crosses into the model itself.

This is Strategy C from the previous step, and it is the one almost everybody ships.

Imagine the feature matrix: age, sex, eGFR. There is no race column. That looks clean. It is easy to document. It is easy to defend.

The model never sees race but upstream, the eGFR already did.

eGFR is not a raw lab value. It is a derived feature — a number the equation produced — and that equation used race. Even if race has left the feature matrix, race-conditioned information can still arrive through that derived variable.

That is the broader problem: information introduced upstream can travel downstream even when the original variable is gone.

We test that directly. We train the same model twice. Neither run has a race column. Same patients, same labels, same split, same seed. The only thing that changes is which equation filled the eGFR column: CKD-EPI 2009 (with the race coefficient) versus CKD-EPI 2021 (race-free).

This example simulates common practice across clinical AI. Teams ship models they say do not use race, then feed them a number an equation already wrote with race in it. Here that number is eGFR. In pulmonary models it is a race-adjusted spirometry percent. In obstetrics it is the recently updated VBAC score (also updated in 2021). The race column is gone. The coefficient is not.

We already know what that change does to the cohort. At the 60 threshold, 2.41% of Black adults move into CKD.

The question now is different. If those adults crossed into CKD, does AUROC stay the same?

AUROC (Area Under the Receiver Operating Characteristic Curve) measures how well a model ranks people by risk.

Here is the simplest way to think about it: pick one adult who died and one who did not. If the model gives the higher risk score to the adult who died, it ranked that pair correctly.

AUROC summarizes how often the model gets that ranking right across many possible pairs. An AUROC of 0.50 is no better than chance. An AUROC of 1.00 means every pair is ranked correctly.

But AUROC does not ask whether someone falls above or below a particular clinical cutoff. It does not see the 60 line. A patient’s eGFR can move across that line — changing how they are classified or treated — while the model’s overall ranking, and therefore its AUROC, barely changes.

The number does not move

Equation	Global AUROC	Worst group by race
CKD-EPI 2009 (race coefficient)	0.901	—
CKD-EPI 2021 (race-free)	0.902	0.881 (Black)

The difference is 0.001. Swapping the equation that reclassified 2.41% of Black adults into chronic kidney disease moves the headline metric by essentially nothing.

Audited by race, this model passes every floor it is measured against.

Why it does not move, and why that is not a bug in your code

AUROC is rank-based. It asks how often the model scores a patient who died above one who did not. CKD-EPI 2021 is a full refit, not a scalar of 2009, so the two equations are not a monotonic transform of each other. Both still decrease in creatinine; within a race group the ranking is largely preserved because the race coefficient is a level effect. It moves where a patient sits relative to a fixed line. Discrimination metrics do not have a fixed line, so they cannot see it.

What this step does not prove

By the metrics we chose, the model passes.

Both race groups clear the worst-group AUROC floor, the gap from global performance is small, and the headline metric barely changes. A model card built from these results could look completely reassuring.

But a clean audit is not the same thing as evidence that no one was harmed.

AUROC tells us whether the model ranks patients well. It does not tell us whether a change upstream caused particular patients to cross a clinically meaningful threshold.

That distinction matters here and it is chronicly overlooked. The metric stayed stable even though some patients changed classification.

So the lesson is not that AUROC failed.

AUROC answered the question we asked. The problem is that we asked a question that could not see the harm.

The next step changes the audit question. Instead of asking,

Does the model still discriminate well?

we ask,

Who crosses the decision boundary when the upstream measurement changes?