On overconfidence

A guest post by Sam Atis

I’m in need of a break this week, with work ratcheting up before Christmas and a long existing piece to finish. But I never like to leave you without writing. This week I'm delighted to feature a guest post from Sam Atis, who writes the excellent

Samstack

. It’s full of crunchy data and detail, all the stuff we don’t normally get here at Stiff Upper Quip. Enjoy.

On Overconfidence

In February last year, I started working for a tiny non-profit that tried to look at the evidence around the effectiveness of protests, to see if they might be a useful tool to try and make the world better.

I can’t give you a clear answer on exactly how effective protests are. What I can tell you is that ‘Are protests effective?’ is an insanely difficult question to answer. There are many, many, many considerations. Which protests are we talking about specifically? What do we mean by ‘effective’? If a protest group falls apart due to in-fighting but was successful in getting people to pay attention to the issue they’re protesting about, are they effective or ineffective? What if there’s a group of protesters who everyone seems to hate, but they’re able to make the headlines a lot? What about a group that achieves their policy goal but the policy ends up not doing what they thought it would?

Then you get to the evidence: How should we weigh one very-well designed study pointing in one direction against several weaker studies pointing in the opposite direction? How do we figure out if these studies have large enough sample sizes? What do we do about nerdy technical difficulties like the problem of multiple comparisons? 

Tricky indeed. Yet what’s fascinating to me is that these weren’t actually tricky questions at all! In fact, it’s extraordinarily easy to figure out whether protests are effective in achieving their aims. Or so I’ve been told. People sympathetic to climate protesters let me know that protests are absolutely, definitely effective! 

Literally every single right you have is because of activism. Women’s rights? Suffragettes. Racial equality? Civil Rights movement. The weekend? Trade union campaigners. The EU’s ‘Right to be Forgotten’? No clue, but presumably activists of some description. And haven’t you seen the charts showing that concern about climate change massively increased in the aftermath of climate protests?

There we go, solved. Until you meet someone who is extremely confident that protests have massively backfired and harmed their cause. Activists from Extinction Rebellion are incredibly unpopular. Didn’t you hear about Just Stop Oil blocking an ambulance from reaching an emergency? Or what about the fact that the majority of people are opposed to protesters disrupting roads and public transport; how could a group possibly be successful if they piss everyone off?

People are overconfident. And on bloody difficult questions, people are bloody overconfident. 

There’s a fun study that I heard about from Adam Mastrioanni (I’m paraphrasing here) that offers a neat demonstration of this. Let me ask you a question: Do you know how a toilet works? Most people reply ‘of course I know how a toilet works, do you think I’m a moron?’. But then if you ask them to explain exactly how a flush toilet works, they start to squirm. Try to write it out, step by step, and tell me specifically how using the toilet handle leads to the toilet flushing and the bowl refilling. Why doesn’t it overfill? If the flush handle stops working, what’s the probable cause? 

Most people don’t know, even though they claim they do. Unless they’re asked to explain in detail, at which point they flail around and blurt out rubbish until they’re forced to sheepishly admit that, in fact, they do not know how a toilet works.

Here’s another example. When we say someone is ‘well-calibrated’, we mean that when they say that there’s an 80% chance of something happening, we would in fact find that the events did occur 80% of the time if we asked them to predict lots and lots of things. So, what happens when you do actually ask people to predict lots and lots of things? If you’re smart enough to have read this much of the article (and are able to read the title of the piece), you can probably guess. They’re poorly calibrated, because they’re overconfident. Here’s a quote from a famous study that examined calibration:

The primary conclusion of this review is that people tend to be overconfident, that is, they exaggerate the extent to which what they know is correct … We see that when people should be right 70% of the time, their "hit rate" is only 60%; when they are 90% certain, they are only 75% right; and so on.

I must confess to having something of an ulterior motive in writing this post. 

While it may seem as though I am presenting some evidence that the morons who took part in this study were overconfident, or the morons who didn’t know how a toilet worked were overconfident, or the morons who thought they knew about protest were overconfident, that’s not what I’m actually trying to do. My real aim is to convince you that we’re all morons who are moronically overconfident.

You probably haven’t heard of the Dunning-Kruger effect before. Just joking! Of course you’ve heard of the Dunning-Kruger effect, and you know perfectly well that it suggests that people who are ignorant are, oddly enough, more confident than people who are knowledgeable. You may have seen this type of thing before:

The problem is that this isn’t what Dunning and Kruger found at all! For one thing, their paper didn’t have much to do with ignorance or knowledge, it was actually a study that looked at ability tests for different skills. 

In one example from the paper, students from Cornell University were asked to rate different jokes by how funny they were. Then they were asked to estimate their ability to recognise what’s funny in comparison to an average Cornell student. The jokes had been judged by professional comedians who gave a supposedly objective rating of how funny the jokes were. What was the finding? Well, it was that the people who were the least good at rating jokes accurately believed themselves to be better than they actually were (as can be seen below).

Still, do you notice something here? It’s not the case that people who did worst on the test give themselves better self-evaluations than the people who did best on the test. They’re not more confident than the top quartile, they’re in fact less confident. It’s true that they overestimate their ability, but the finding here is quite different to the way it’s often interpreted. 

In a pleasingly ironic turn of events, people are hugely overconfident about how much they know about the most famous study on overconfidence. (This is all without mentioning the fact that the study results might not even be particularly reliable.)

What’s the takeaway here? 

It would be faintly ridiculous if it were ‘we can be extremely confident that most people are overconfident’, so it’s probably not that. I guess it’s something more like: We should have some epistemic humility about the things we feel most certain about. Lots of smart people have been very certain about things they thought were very simple and turned out to be very wrong. You should lean into the uncertainty. Or least, you probably should. I’m not certain.

Sam Atis writes about social science and effective altruism at samstack.io.