The Second Failure
Step 8Live-room allocation: 8:00
Open in Colab — 02_act2.ipynb, opens in a new tab
An aggregate can look complete while hiding who is missing from the story.
Nothing changes about the model in this step.
Same model. Same predictions. Same held-out test set as Act II.
The only thing we change is how we look at the results.
Instead of asking how the model performs across everyone together, we separate the same predictions by insurance status.
Disaggregate by insurance
Section titled “Disaggregate by insurance”| Group | n | AUROC | 95% CI | ECE |
|---|---|---|---|---|
| Uninsured | 970 | 0.776 | 0.729–0.826 | 0.036 |
| Insured | 3,467 | 0.911 | 0.899–0.921 | 0.028 |
Globally, the model’s AUROC is 0.902.
Among uninsured adults, it falls to 0.776.
That is a drop of 0.125 from the headline number.
The global AUROC was not wrong.
It was incomplete.
Most patients in this test set are insured, so the global metric is driven primarily by performance in that larger group. Once the predictions are disaggregated, the weaker performance among uninsured adults becomes visible.
Which rule failed?
Section titled “Which rule failed?”DON'T SHIP — worst is 0.125 below global
The gate checks three things.
First, does every group clear the minimum AUROC floor?
The floor is 0.65. The uninsured group scores 0.776, so that rule passes.
Second, is calibration error below the limit?
The maximum allowed ECE is 0.10, and the worst observed ECE is 0.036. That rule passes too.
The failure comes from the third rule:
How far can any subgroup fall below global performance?
The maximum allowed drop is 0.10. The observed drop is 0.125.
That crosses the limit, so the gate fires.
A gate that checked only whether every subgroup cleared an absolute minimum would have missed this.
Two different failure modes
Section titled “Two different failure modes”The race audit and the insurance audit fail for different reasons.
By race, AUROC stayed stable while patients crossed clinical thresholds. The problem appeared downstream, at the decision.
By insurance, the performance gap is visible in AUROC itself.
The metric is capable of seeing this failure. The aggregate is what hides it.
Sometimes the metric cannot measure the consequence you care about.
Sometimes the metric can measure it perfectly well, but averaging across the population conceals where performance is worse.
Those require different audit strategies.
Someone had to name the group
Section titled “Someone had to name the group”There is one more problem.
We only found this gap because we chose to examine insurance status.
Race was already visible because the equation explicitly used it. Insurance was not.
Nothing in the model card, the global AUROC, or the training pipeline tells us which subgroup to inspect next.
Someone has to decide what dimensions might matter. And if that decision depends only on what the team already thought to check, important failures can remain invisible.
The next step asks how to search more systematically.