Overconfidence: When Excessive Self-Belief Wins

People are overconfident, on the whole. Most of us believe we are more intelligent, more skilful or better drivers than the average person. But we can't all be right.

Such beliefs are not harmless. Overconfidence - and the self-deception that drives this excessive self-belief - can contribute to extreme risk-taking , failed businesses, disastrous investments, dangerous driving and political miscalculations. Why then have so many of us retained this trait?

According to the late evolutionary biologist Robert Trivers , self-deception evolved in humans because it also helps us deceive others more effectively. He argued that when people genuinely believe in a lie or falsehood, they suppress the subtle cues that betray deception - such as nervousness, hesitation, vocal strain or a forced expression.

But we think something is missing from Trivers's account. When overconfidence is such a common trait, why should anyone believe your hype simply because you believe it yourself? This is the question our latest research set out to answer.

Our approach was based on the handicap principle developed by Israeli biologists Amotz and Avishag Zahavi. The classic example is the peacock's tail, which demands enormous metabolic energy to grow and maintain, and makes flying, running and escaping from predators harder.

Only an unusually healthy, capable bird can grow such plumage and survive while dragging it around. Because the tail is so costly, it works as an "honest signal" of these attributes. Our claim is that self-belief is a peacock's tail for people, signalling attributes that are hard to observe directly.

The signal works because the cost of high self-belief is lower for more capable people. Think of it this way: if a complete novice driver has a self-belief rating of ten out of ten, they are very likely to crash with potentially disastrous results. A near-expert driver with extreme self-belief is, of course, much less likely to crash.

Moreover, the benefit of high self-belief is generally greater for more able people. While self-belief opens doors, ability dictates how far someone goes as a result.

But because self-belief is an effective signal of ability, a confidence "arms race" ensues that culminates in the widespread overconfidence among people that we described earlier. Beliefs become excessive - yet even then, we argue that they remain informative. And that's why overconfidence is such a common condition.

Reappraising the Dunning-Kruger effect

In an influential 1999 paper , US psychologists David Dunning and Justin Kruger stated that the least competent people tend to be the most confident.

According to their original interpretation, most people who are bad at a task lack the awareness needed to recognise how badly they are doing it. They are "unskilled and unaware of it".

The original study followed a simple design. Participants completed tests of humour, grammar or logical reasoning. Before receiving feedback, they estimated how well they had performed. Researchers then examined whether the expectations of high scorers were more accurate than those of low scorers - and found they were. Less skilled participants overestimated their performance more egregiously.

However, there is a fundamental flaw in this methodology that even Dunning and Kruger recognised . Test performance depends heavily on random factors like luck. As forecasts cannot take these factors into account, low scorers appear more overconfident to a large degree because they were simply unlucky on the test.

To overcome this statistical illusion, the procedure can be flipped. Instead of sorting people by their test scores, they are grouped by their expectations. Asking whether those with high expectations overestimate by more or less than those with low expectations eliminates the "unlucky" measurement bias.

For our study , we revisited data from one of the largest modern replications of the original Dunning-Kruger experiments, involving several thousand participants. Building on the flipping logic to treat randomness in both test scores and predictions equally, we found that those with high expectations performed better and overall exhibit greater overconfidence - exactly as the signalling theory implies.

Loss aversion - a complicating factor?

The same signalling logic, we think, also illuminates another common human trait: loss aversion . It is sensible to dislike losses, but the evidence shows we dislike them too much.

While loss aversion is easily confused with standard risk aversion , the two are fundamentally different. Risk aversion arises because gaining £300 isn't twice as joyful as gaining £150, so it is rational to prefer a guaranteed £150 over a 50-50 opportunity to gain £300 or nothing.

The evidence is, though, that many choices cannot be explained by this framework . Instead, it is loss aversion, our excessive aversion to losing, that is now the dominant explanation for our excessive rejection of risks.

Overweighting the downside pulls in the opposite direction to overconfidence: it makes us cautious, wary of risky investments, speculative trades and uncertain ventures.

At first glance, this looks like a problem for the whole story. If loss aversion makes us too cautious to ever act on our confident claims, then surely big talk would cost - and mean - nothing? In which case, why has caution not simply cancelled out overconfidence, leaving us with accurate beliefs and no signal value at all?

We argue there are two reasons it has not. The first is that caution cannot be tuned precisely enough to neutralise overconfidence. Doing so would require a different, finely judged level of loss aversion for every situation we face - entrepreneurship, trading, courtship, competition - and that degree of case-by-case calibration is cognitively far too demanding.

So, on a population-wide level, our estimates of caution offset overconfidence only imperfectly. While this balance varies significantly from person to person, enough average cost survives across the population for the signal to remain credible.

In addition, while confidence must be displayed to be effective, there is obviously no need to brag about your timidity - and no gain to be had from doing so. This means that overconfidence remains a costly, and thus credible, signal for many people.

So, we think former US president Franklin D. Roosevelt had a point when he said of his army general, Douglas MacArthur: "Never underestimate a man who overestimates himself."

The authors do not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and have disclosed no relevant affiliations beyond their academic appointment.