Cognitive Reflection Test: 3 Questions, Answers and Meaning

Facebook ↗WhatsApp ↗

Originally published 2017-10-14.

The Three-Question Cognitive Reflection Test

The Cognitive Reflection Test (CRT) is a three-question puzzle set created by psychologist Shane Frederick, then at the MIT Sloan School of Management, and published in the Journal of Economic Perspectives in 2005. Each question has an obvious answer that pops into your head immediately, and that answer is wrong. The test measures how readily you stop, check and override that first impulse.

It is often shared online as "the shortest IQ test ever", but that label is misleading. The CRT is a short measure of reflective thinking that correlates moderately with intelligence tests; it does not produce an IQ score. It is still a genuinely hard little test: in Frederick's data from 3,428 people, only 17% answered all three questions correctly. Try it before reading the answers.

The three questions

  1. A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost?
  2. If it takes 5 machines 5 minutes to make 5 widgets, how long would it take 100 machines to make 100 widgets?
  3. In a lake there is a patch of lily pads. Every day the patch doubles in size. If it takes 48 days for the patch to cover the entire lake, how long would it take for the patch to cover half of the lake?

Take a moment, write your answers down, and then scroll on.

…

The answers, and why the first guess fails

1. The ball costs 5 cents

Most people say 10 cents. Check it: if the ball were 10 cents, the bat would be $1.10 and together they would cost $1.20. Call the ball x. The bat is x + $1.00, so x + x + $1.00 = $1.10, which gives 2x = $0.10 and x = $0.05. A 5-cent ball and a $1.05 bat add up to $1.10, and the bat is exactly a dollar more.

2. It takes 5 minutes

The tempting answer is 100 minutes, because the pattern "5, 5, 5" seems to continue as "100, 100, 100". But the question tells you each machine takes 5 minutes to make one widget. A hundred machines working side by side each make one widget in the same 5 minutes.

3. It takes 47 days

Many people halve 48 and say 24. Because the patch doubles every day, it must have covered half the lake on the day before it covered the whole lake. That is day 47. Exponential growth is notoriously hard for intuition: for most of the period the patch looks small, then it seems to explode at the end.

What the test actually measures

Psychologists often describe thinking in terms of two modes. "System 1" is fast, automatic and intuitive. "System 2" is slower, deliberate and effortful. The labels come from researchers Keith Stanovich and Richard West and were popularised by Daniel Kahneman in his book Thinking, Fast and Slow.

The CRT is designed so that System 1 serves up a confident wrong answer. Getting the right answer does not need advanced maths; it needs the habit of doubting yourself for a few seconds. That is why the test interests economists and psychologists: people who score higher tend to make different choices about money, time and risk.

In Frederick's samples, CRT scores correlated with other cognitive measures at around 0.44 for the SAT, 0.46 for the ACT and 0.43 for the Wonderlic personnel test. Those are real but moderate links. They suggest the CRT overlaps with general reasoning ability while also capturing something more specific: the willingness to check an intuition.

How different groups scored

Frederick ran the test across 35 studies between 2003 and 2005, mostly with university students. The average score was 1.24 out of 3. Overall, 33% got none right, 28% got one, 23% got two and 17% got all three.

Results varied strongly by setting. The share of people with a perfect score in some of the groups was:

  • MIT: 48% (average score 2.18)
  • Princeton: 26%
  • Carnegie Mellon: 25%
  • Harvard: 20%
  • University of Michigan, Ann Arbor: 14%
  • Online participants: 13%
  • University of Toledo: 5%

One group stands out for fun: people surveyed at a fireworks picnic in Boston did slightly better than the Harvard students on average. Contrary to a popular version of this story, Yale does not appear in the published table, so claims about "Yale and Harvard students failing" overstate what the study showed.

What high and low scorers did differently

The more interesting part of the research is what the scores predicted. Compared with low scorers, people who answered all three correctly were more patient with money over short time frames: they were more willing to wait for a larger reward. Low scorers were willing to pay about twice as much for overnight delivery of a book they wanted ($4.54 on average versus $2.18).

The bat-and-ball question also revealed a confidence gap. People who answered 10 cents estimated that about 92% of others would get it right; those who solved it guessed 62%. In other words, the people who missed it tended to think it was easy. In a Princeton sample, respondents who gave the 10-cent answer were also measurably less patient in money choices than those who said 5 cents.

Risk attitudes differed too. High scorers were more willing to take gambles in the domain of gains and less likely to gamble to avoid losses, while low scorers showed the classic pattern described by prospect theory. Frederick also found that men scored higher than women on average, a gap that remained after controlling for SAT maths scores, and that women's mistakes were more often the intuitive wrong answer.

Limits and common myths

  • "It's an IQ test." It is not standardised as one and was never designed to meet the professional standards for psychological testing. Its correlation with intelligence measures is moderate.
  • "Answering fast proves you are a genius." Nothing in the original research scores speed. If anything, the point of the test is that slowing down helps.
  • "Three questions are enough to judge a person." Three items are a very small sample. One lucky or unlucky moment changes your score by a third.
  • "Everyone sees it fresh." The puzzles became famous. Researchers estimate that roughly 44% to 51% of study participants have seen them before, and people who have tend to score higher. That is one reason psychologists developed longer and alternative versions, such as an expanded test by Toplak, West and Stanovich (2014) and an alternate form by Thomson and Oppenheimer (2016).

How to use the lesson in everyday life

You do not need to ace the CRT to benefit from its message. A few habits help whenever a question seems suspiciously easy:

  • Plug your answer back in. Checking 10 cents against the original conditions exposes the error in seconds.
  • Watch for patterns that feel too neat. "5, 5, 5 so 100, 100, 100" is a pattern-matching shortcut, not reasoning.
  • Think backwards with growth. For anything that doubles, such as debts, viral posts or infections, ask what it looked like one step earlier.
  • Notice confidence. A strong feeling of certainty is not evidence. The research shows the most confident people were often the ones who got it wrong.

If you got all three right, congratulations: you are in the same company as about half of the MIT students Frederick tested. If you did not, you have plenty of good company too, and you now know the trick that matters most, which is simply to pause.

Sources: Shane Frederick, "Cognitive Reflection and Decision Making", Journal of Economic Perspectives (2005), PDF; Cognitive reflection test (Wikipedia); originally shared via IFLScience.

COMMENTS