Credence – using subjective probabilities to express belief strengths

There are surprisingly many impediments to becoming comfortable making personal use of subjective probabilities, or “credences”: some conceptual, some intuitive, and some social. However, Phillip Tetlock has found that thinking in probabilities is essential to being a Superforcaster, so it is perhaps a skill and tendency worth cultivating on purpose.

To help cultivate more fluency with thinking in probabilities, back in 2012 some friends and I created the Credence Calibration Game to help people practice reporting credences and calibrating them over time. This post is a basic introduction to being comfortable assigning credences to things, and is for anyone who is unfamiliar or uncomfortable with doing that.

Note: Thinking in probabilities is a prerequisite skill to Bayesian updating, which is explained reasonably well in these tutorials.

Introduction

When people say “I’m 90% sure Joe is at the party”, the 90% means very different things for different people. In fact, in some experiments, when we say we’re 99% sure, we’re only right 40% of the time! This is called the overconfidence effect. It would be nice to fix that discrepancy, so we’d know just how seriously to take each other when we say “90% sure”.

Fixing the discrepancy is called calibration, and can be achieved through practice, for example by playing the Credence Game. But for many people, uttering numbers like 90% sure feels meaningless, arbitrary, unprincipled, and even causes anxiety, so they can’t even start practicing. So, for the sake of improving communication and group decision-making practices everywhere, I want to help people overcome this anxiety. After calibration practice — after practice converting our internal cues about how much to trust ourselves into percentages and getting suitable feedback — these numbers stop being arbitrary and start being informative, to ourselves and to others.

A definition of “90% sure”

In Bayesian statistics, the 90% in “I’m 90% sure” is called your credence, and the phrase

“I’m 90% sure that Joe is at the party”

is defined to mean roughly that

“I’d rather bet that Joe is at the party than bet on an 89%-biased roulette wheel, and I’d rather bet on a 91%-biased roulette wheel than bet that Joe is at the party.”

Strictly speaking, this is the definition of being between 89% and 91% sure, but you get the idea.

A cool thing about this definition is that it is not asking “how much are you willing to bet?”. We all have different incomes, and different levels of desire for security, which could all affect how much money we’re willing to bet, even on a 90%-biased roulette wheel. (By the way, in VNM utility theory, these is nothing considered irrational about this!) By making the question a choice between bets rather than between amounts of money, we get a clearer picture of how sure we are, in a way that’s independent of our other preferences.

Making bets more intuitive

Imagine a physical source of randomness to compare your credences with. When I say a “imagine a 90%-biased roulette wheel”, I mean a roulette wheel where 90/100 slots have “win” written on them, and the others say “lose”. If you don’t like those, you could imagine a jar with 90/100 marbles in it with the word “win” written on them, or a coin that lands with the word “win” face up 90% of the time. The important thing is to have something of this sort that you can personally comfortably visualize.
Give a range of values that you think you’d settle between if you thought longer and harder. If 89% and 91% seem too precise to distinguish between as a human, try using 85%-95%, or 80%-99%. Giving yourself and others some description of where your mind changes is more helpful than giving no description.
Imagine repetitions of the thing you’re assigning a credence to. If this party happened 100 times, how many times do you think Joe would show up? It’s a silly question, but for some people it helps to think about a non-binary answer.
Flag when you anticipate updates. You can be 50% sure that a coin will land heads, and before the actual event, you can be quite confident that further investigation will not change your mind. By contrast, you can be 50% sure Joe will be at the party, but expect to change that to 10% or 90% very soon because you’re about to call him and ask. These can feel like very different kinds of “50%”, and they are: in one case, you’re expecting your credence to stay the same over time, and in the other, you’re expecting it to change. Some people call this “meta uncertainty”, but more precisely it’s uncertainty about your future uncertainty or about the uncertainty of other better-informed people.

Making bets less awkward

If imagining making a bet at all feels strange or uncomfortable, see if these points help:

Imagine not being embarrassed. If you feel anxiety when you imagine betting, as I and many others I’ve talked to have, it may be because losing bets can be socially embarrassing. To help alleviate that, try imagining the bet is a secret that no one will know about except you.
Share your subjective probabilities with nice people who won’t judge you harshly if they disagree with you, as a warm up to sharing them more.
Don’t worry if betting feels immoral. Remember that asking yourself this question is just imaginary, and shouldn’t lead to you developing a gambling addiction. You can also try imagining that the pay-off for winning is a very large donation to a charity of your choice to avoid feeling greedy.

Assessing your credence

Once you’re comfortable imagining yourself betting in some way — and I encourage everyone to find a way to do that! — you can assess your credence of real-life statements like “Joe is at the party” as follows:

Think of a number Y, say Y=51%,
Ask yourself if you’d rather bet that Joe is at the party or bet on a Y%-biased marble jar. Say when Y=51% you prefer to bet on Joe. Then,
Gradually increase Y in your mind, until you feel like betting on the wheel instead of Joe, and then report roughly the value of Y where you changed your mind.

Not assuming others think this way.

I’m not at all suggesting that you start assuming other people are thinking like this when they say “90% sure”… because they aren’t! But if someone says “90% subjective credence,” you can be pretty credent that this is what they’re talking about! And in my experience, credence is a very handy notion to have around in conversations. Giving a subjective probability is like telling people exactly how you’d put your money where your mouth is, if you had to (and if it was a secret, and for a good cause, and not addictive, etc.).

There terms “credence” and “subjective probability” are often used interchangeably, but there is some contention in statistics over what “probability” should mean, so I tend to say credence a lot to clear that up. Confusingly, “90% confident” in statistics usually means something closer to being “at least 90% sure”, so I try not to say “90% confident” except to say “at least 90% confident”, so everyone gets roughly the same idea of what I mean.

Calibration

Now that we’ve decided what “90% sure” means, it would be nice if it matched up with other related probabilities, like our frequency of success on answering questions. That way, when you said you were 90% sure of something, other people would know that you had a 90% success rate among similar instances of sureness. That’s what calibration practice is for 🙂