Fluency feels true

Lesson 6 · Thinking Fast and Slow × AI · about 10 minutes

Mission: stop reading smooth, confident AI prose as evidence that it is accurate.

The core idea

Kahneman describes cognitive ease: when information is easy to process, we tend to feel it is familiar, true, and safe. Repetition makes a statement easier to process, and that ease is read as truth. This is the illusory truth effect, and it is usually explained as fluency (see the cognitive ease discussion in Part II of the book).

AI output is consistently fluent. Clean grammar, tidy structure, and a confident tone all increase ease. None of that is evidence of accuracy, and the model can produce a fluent falsehood as easily as a fluent truth.

What the research shows

A recent preprint found that without credibility signals, people could not reliably distinguish high-fluency AI hallucinations from accurate text (arXiv 2609.03460). The term "fluency trap" appears in some AI papers, but it is informal, so treat it as shorthand, not an established construct. Direct tests of AI authorship and repetition together are still limited.

Practice

Choose the bias the scenario shows.

1. A well-formatted AI answer includes a citation with an author and journal. It reads so cleanly that you assume the paper exists and move on.
Correct. The polished citation looked real, so you stopped checking.
Not quite. The smooth presentation lowered your scrutiny. That is fluency at work.
2. You see the same claim in three AI answers across different chats. You now believe it, though you never checked a source.
Correct. Repetition across responses made the claim feel more true.
Not quite. Repetition increases processing ease, which the mind reads as truth. That is the illusory truth effect.
3. You want to break the fluency effect on an AI answer. Which step does that most reliably?
Correct. A single source check breaks the spell of fluency.
Not quite. Only a check against a source you control separates fluent from accurate.

Takeaway

Smoothness is a property of the text, not of the facts. Treat any specific claim, name, number, or citation as unverified until you have checked it outside the chat.

Try it this week: ask the AI for three sources on a topic you know, then open each one. Count how many exist as described.

Sources

Questions? Stuck on any idea? Ask your teacher in this conversation. Previous: Lesson 0005. Next: Lesson 0007, confirmation and sycophancy.
← Previous Next →