Overconfidence and calibration

Lesson 8 · Thinking Fast and Slow × AI · about 10 minutes

Mission: match my trust in an AI answer to how often it is actually right.

The core idea

Kahneman's account of overconfidence is that System 1 builds the most coherent story it can from the evidence at hand, and we feel confident in proportion to the coherence, not to the amount of evidence. Calibration is the match between confidence and accuracy. Good calibration means being right about 80 percent of the time when you say 80 percent (see the overconfidence chapters in Part III of the book).

AI can distort calibration in both directions. Long, detailed explanations raise confidence, and so does a smooth, unhedged tone.

What the research shows

In a study of 301 participants, longer, more elaborate AI explanations increased confidence in the answers without improving people's ability to tell correct answers from incorrect ones. Showing uncertainty information raised accuracy without raising confidence, which suggests uncertainty displays can curb overconfidence (arXiv 2603.29681, a preprint, verify before citing). A separate evaluation of interventions found that a reliance disclaimer was the only one that improved appropriate reliance and calibration, and only on one task (To Rely or Not to Rely, arXiv 2412.15584, preprint).

Practice

Choose the bias the scenario shows.

1. The AI gives a long, detailed explanation for a tax position. You feel more sure it is right, though you have not checked the rule.
Correct. Length and detail raised your confidence without raising the answer's accuracy.
Not quite. Your confidence rose with the length of the explanation, not with better evidence. That is overconfidence.
2. An AI answer states a range and names what it is unsure about. Research suggests this does what for your judgment?
Correct. Stated uncertainty helps you separate reliable answers from shaky ones.
Not quite. Uncertainty cues tend to improve judgment by lowering confidence where it is not earned.
3. You want to know how much weight an AI answer can bear. Which follow-up is most useful?
Correct. What would change its answer exposes the assumptions the answer rests on.
Not quite. Asking what would change the answer surfaces the assumptions, which is the calibration check.

Takeaway

Confidence is a feeling produced by coherence. Ask the AI for uncertainty, the assumptions behind its answer, and what would change its mind, then set your trust accordingly.

Try it this week: for three AI answers, write down your confidence from 0 to 100 before checking. Then check and see whether your confidence matched the outcome.

Sources

Questions? Stuck on any idea? Ask your teacher in this conversation. Previous: Lesson 0007. Next: Lesson 0009, substitution.
← Previous Next →