What to do Monday
Step 12Live-room allocation: 2:00
One model. One question. This week. Illustration: Brian Stauffer.
You do not need to rebuild your evaluation pipeline on Monday.
Take one model you already have and ask it one question you are not currently asking.
What to do this week
Section titled “What to do this week”Pick the model you would be least comfortable defending to someone affected by it.
Take a metric you already report — AUROC, accuracy, calibration error, false-negative rate — and split it by one axis you do not currently report.
Choose an axis that a person affected by the model might reasonably ask about.
Then look.
If the global number still holds, you learned something.
If it does not, you found an audit question worth pursuing.
And if the field is not in your data, that is a result too.
You cannot audit a difference you never collected the information to examine.
Pre-commit before you look
Section titled “Pre-commit before you look”Write down:
- the metric
- the subgroup
- why it matters
- what would trigger review
Then run the audit.
The goal is not to find a failure.
The goal is to know what you would do if one appears.
If the result looks clean
Section titled “If the result looks clean”Do not stop at:
the metric passed.
Ask:
Was this metric capable of seeing the consequence I care about?
A reassuring number only matters when the metric is aligned with the decision you are trying to protect.
Keep going
Section titled “Keep going”Everything we did today was still largely human-directed.
You chose where to search.
You interpreted the evidence.
You decided what triggered review.
The next question is what happens when the search itself becomes agentic.
Could an agent choose which subgroups to inspect, run the analysis, track how much it searched, and know when the evidence is not strong enough to proceed?
That is the next design problem:
search → evaluate → gate
The first two can increasingly be automated.
The gate still has to encode what evidence is required, when uncertainty is too high, and when a human has to intervene.
Your NVIDIA DLI credits give you a practical place to continue.
Build a Deep Research Agent is a natural next step for learning how agents plan, retrieve information, use tools, and execute multi-step work.
Bring one principle from this workshop with you:
Do not just tell an agent what to find. Define what it is allowed to conclude.
Monday
Section titled “Monday”Take one model.
Choose one consequence.
Choose one subgroup your current evaluation does not show you.
Write down what would make you stop.
Then look.
One model. One question. This week.