The Hidden Harm Challenge

Step 9·Live-room allocation: 19:00 min

▶Open in Colab — 03_hidden_harm.ipynb, opens in a new tab

You choose where to look. What the light does not hit stays unmeasured.

Until now, every subgroup was chosen for you.

We looked at race because the equation explicitly used race.

We looked at insurance because the instructor thought to ask whether performance changed by insurance status.

But a real audit rarely arrives with the important subgroup already circled.

So now you choose where to look.

You control the search

Open 03_hidden_harm.ipynb on a Colab T4:

Runtime → Change runtime type → T4 GPU

Then find the block marked EDIT HERE:

AXES = ["race", "insurance"]
MAX_DEPTH = 2
MIN_N = 30
ALPHA = 0.05

Those four lines define the search.

• AXES controls which characteristics you examine.
• MAX_DEPTH = 1 tests each characteristic separately.
• MAX_DEPTH = 2 also tests their intersections.
• MIN_N determines how large a subgroup must be before we evaluate it.
• ALPHA sets the significance level after accounting for the full search.

Available axes include race, insurance, diabetic, and female.

Change the search. Re-run it. Watch what survives.

The result is not just the winner

Run the default search.

It considers 8 possible subgroups across race, insurance.

But those groups do not all make it to the end.

subgroups considered	8
large enough to audit	8
survive correction	3
removed for small sample size	0
removed after multiplicity correction	5

The most important number in that table may be the one that does not survive.

5 subgroups looked notable before correction.

If any one of those had been the only subgroup we tested, it might have looked worth reporting.

But it was not the only subgroup we tested.

We looked at 8 of them.

That changes the standard of evidence.

Looking in more places has a cost

If you test enough subgroups, eventually one will look unusually good or bad just by chance.

That is the multiple-comparisons problem.

This notebook accounts for it directly.

During each permutation, it does not ask:

How extreme was this one subgroup?

It asks:

How extreme was the worst subgroup anywhere in the entire search?

Each observed subgroup is then compared against that distribution of worst-case results.

That means the search pays for every place you chose to look.

Add more axes and the correction becomes harder to survive.

Reduce MAX_DEPTH and the search family becomes smaller.

The patients did not change.

The predictions did not change.

What changed was how many opportunities you gave yourself to find something extreme.

For the default search, the null minimum is 0.8234.

That tells us how low the worst subgroup statistic can fall by chance alone when a search of this size is performed.

The search rediscovers insurance

Run the default configuration and the surviving findings are insurance-related.

insurance=Uninsured returns an AUROC of 0.776.

That is the same failure we found in step 8, but this time we reached it through a systematic search rather than by choosing insurance first.

An intersection can look worse still:

race=Non-Black & insurance=Uninsured

But notice what does not survive.

No race-only finding survives the corrected AUROC search.

That does not contradict Act I.

It tells us something more important about the search itself.

A search can only find what its metric can see

The race-related harm earlier in the workshop happened at a clinical threshold.

Patients changed CKD classification and prescribing tier while their relative risk ranking barely changed.

AUROC is a ranking metric.

So an AUROC-based subgroup search is not designed to detect that kind of harm.

Insurance is different.

There, model discrimination itself becomes worse for a subgroup, so AUROC is exactly the kind of metric that can expose the problem.

This gives us another audit lesson:

A broader search does not rescue the wrong metric.

You can search every subgroup in the dataset and still miss a real consequence if the statistic you are searching cannot represent it.

Silence is not proof of safety.

Read excess, not just AUROC

There is one more trap.

A subgroup can have a poor AUROC simply because it is small.

A subgroup with very few positive outcomes can also produce an unstable AUROC.

Neither observation, by itself, proves that the model performs systematically worse for that group.

So the notebook does not treat raw AUROC as sufficient evidence.

For each subgroup, it constructs a reference matched on two things:

• subgroup size
• outcome prevalence

The reference asks:

What would performance look like for a group with the same statistical shape, drawn from everyone else?

The notebook then reports the difference as excess.

Negative excess means the subgroup performs worse than comparable groups would lead us to expect.

A value near zero means the strange-looking AUROC can largely be explained by the subgroup’s size or outcome mix.

So when you inspect the lattice, do not ask only:

Which group has the lowest AUROC?

Ask:

Which group performs worse than comparable groups should?

That distinction prevents a small or unusually composed subgroup from being mistaken for evidence of differential model performance.

When the search gets too thin

Now make the search more aggressive.

Add dimensions.

Increase depth.

Lower MIN_N.

Eventually you can create dozens of small intersectional groups.

Some will be suppressed because there are too few observations to audit them responsibly.

That is not the search refusing to cooperate.

It is the audit telling you that the data no longer support the question you are asking.

The tempting response is to lower MIN_N until the subgroup appears.

But lowering the minimum sample size does not create evidence.

It removes the rule that was warning you that there was too little evidence to make the comparison reliably.

The notebook says it directly:

The tempting fix is to lower MIN_N until more groups appear. That does not create evidence; it removes the check that was telling you there was none.

An empty result can mean:

the subgroup lattice became too thin to evaluate.

It does not mean:

the model passed.

Your challenge

You have a few minutes.

Start with the default search. Then each challenge is one edit and one run.

Challenge 1: Widen the search

Start with:

AXES = ["race", "insurance"]
MAX_DEPTH = 2
MIN_N = 30
ALPHA = 0.05

Now add diabetic:

AXES = ["race", "insurance", "diabetic"]

Before you run it, make a prediction.

What happens to:

• the number of subgroups considered?
• the number large enough to audit?
• the number that survive correction?
• p_fwer?

Then run the comparison cell.

The data did not change. The model did not change.

You changed how many places you looked.

Challenge 2: Shrink the search

Return to:

AXES = ["race", "insurance"]

Now change:

MAX_DEPTH = 1

With depth 1, the notebook searches each axis separately but does not search their intersections.

Run the comparison cell.

The patients are identical. The predictions are identical. The subgroup AUROCs are identical.

Only the size of the search family changed.

Look for a subgroup whose corrected result changes.

Then ask:

Did the underlying evidence change — or did the number of questions we asked change?

Optional challenge: Make the lattice too thin

If you have time, run the four-axis search as it is already filled in:

AXES = ["race", "insurance", "diabetic", "female"]
MAX_DEPTH = 3
MIN_N = 30

Only a few groups fail the size rule. Read the suppressed rows, then the smallest auditable intervals. A wide interval is also the lattice running out of evidence.

Then run the cell that is already set to:

MIN_N = 5

You will get more auditable cells.

But more cells are not automatically more evidence.

Watch what happens to the null distribution and the multiplicity correction.

You bought more opportunities to find something extreme.

The search charges you for them.

What the challenge is really testing

The goal is not to find the most alarming subgroup in the notebook.

The goal is to notice how quickly the answer can depend on choices made before you ever interpret a result:

• which axes you searched
• how deeply you intersected them
• how small a subgroup you allowed
• which metric you searched
• how many comparisons you made

Those are not implementation details.

They define what the audit is capable of finding and what evidence is required before you call something a finding.

A search that reports only its most extreme result leaves out part of the analysis.

You also need to know how much searching produced that result.

The question to take back to your own model

When you see a subgroup result in a model card, paper, dashboard, or fairness report, ask:

How many other subgroups were examined before this one was reported?

Then ask a second question:

Was the metric being searched capable of detecting the kind of harm we care about?

Those two questions protect against different mistakes.

One protects against discovering a subgroup simply because you searched enough of them.

The other protects against declaring safety because your chosen metric stayed quiet.

A responsible subgroup search reports not only what it found.

It reports how hard it looked and what it was capable of seeing.

← Previous8. The second failure8:00 min

Next →11. The go/no-go checklist3:00 min