# Decision Theory

Everywhere below where a Predictor is mentioned, assume that their predictions are made by scanning your brain to create a highly accurate simulation of you and then observing what this simulation does.

All the below scenarios are one-shot games. Your action now will not influence the future decision problems you end up in.

Newcomb’s Problem
Two boxes: A and B. B contains \$1,000. A contains \$1,000,000 if the Predictor thinks you will take just A, and \$0 otherwise. Do you take just A or both A and B?

Transparent Newcomb, Full Box
Newcomb problem, but you can see that box A contains \$1,000,000. Do you take just A or both A and B?

Transparent Newcomb, Empty Box
Newcomb problem, but you can see that box A contains nothing. Do you take just A or both A and B?

Newcomb with Precommitment
Newcomb’s problem, but you have the ability to irrevocably resolve to take just A in advance of the Predictor’s prediction (which will still be just as good if you do precommit). Should you precommit?

Take Opaque First
Newcomb’s problem, but you have already taken A and it has been removed from the room. Should you now also take B or leave it behind?

Smoking Lesion
Some people have a lesion that causes cancer as well as a strong desire to smoke. Smoking doesn’t cause cancer and you enjoy it. Do you smoke?

Smoking Lesion, Unconscious Inclination
Some people have a lesion that causes cancer as well as a strong unconscious inclination to smoke. Smoking doesn’t cause cancer and you enjoy it. Do you smoke?

Smoking and Appletinis
Drinking a third appletini is the kind of act much more typical of people with addictive personalities, who tend to become smokers. I’d like to drink a third appletini, but I really don’t want to be a smoker. Should I order the appletini?

Expensive Hospital
You just got into an accident which gave you amnesia. You need to choose to be treated at either a cheap hospital or an expensive one. The quality of treatment in the two is the same, but you know that billionaires, due to unconscious habit will be biased towards using the expensive one. Which do you choose?

Rocket Heels and Robots
The world contains robots and humans, and you don’t know which you are. Robots rescue people whenever possible and have rockets in their heels that activate whenever necessary. Your friend falls down a mine shaft and will die soon without robotic assistance. Should you jump in after them?

Death in Damascus
If you and Death are in the same city tomorrow, you die. Death is a perfect predictor, and will come where he predicts you will be. You can stay in Damascus or pay \$1000 to ﬂee to Aleppo. Do you stay or ﬂee?

Psychopath Button
If you press a button, all psychopaths will be killed. Only a psychopath would press such a button. Do you press the button?

Parfit’s Hitchhiker
You are stranded in the desert, running out of water, and soon to die. A Predictor will drive you to town only if they predict you will pay them \$1000 once you get there. You have been brought into town. Do you pay?

XOR Blackmail
An honest predictor sends you this letter: “I sent this letter because I predicted that you have termites iﬀ you won’t send me \$100. Send me \$100.” Do you send the money?

Twin Prisoner’s Dilemma
You are in a prisoner’s dilemma with a twin of yourself. Do you cooperate or defect?

Predictor Extortion
A Predictor approaches you and threatens to torture you unless you hand over \$100. They only approached you because they predicted beforehand that you would hand over the \$100. Do you pay up?

Counterfactual Mugging
Predictor ﬂips coin which lands heads, and approaches you and asks you for \$100. If the coin had landed tails, it would have tortured you if it predicted you wouldn’t give the \$100. Do you give?

Newcomb’s Soda
You have 50% credence that you were given Soda 1, and 50% that you were given Soda 2. Those that had Soda 1 have a strong unconscious inclination to choose chocolate ice cream and will be given \$1,000,000. Those that had Soda 2 have a strong unconscious inclination to choose vanilla ice cream and are given nothing. If you choose vanilla ice cream, you get \$1000. Do you choose chocolate or vanilla ice cream?

Meta-Newcomb Problem
Two boxes: A and B. A contains \$1,000. Box B will contain either nothing or \$1,000,000. What B will contain is (or will be) determined by a Predictor just as in the standard Newcomb problem. Half of the time, the Predictor makes his move before you by predicting what you’ll do. The other half, the Predictor makes his move after you by observing what you do. There is a Metapredictor, who has an excellent track record of predicting Predictor’s choices as well as your own. The Metapredictor informs you that either (1) you choose A and B and Predictor will make his move after you make your choice, or else (2) you choose only B, and Predictor has already made his choice. Do you take only B or both A and B?

# Rationality in the face of improbability

I recently read my favorite Wikipedia article of all time. It’s about a park ranger named Roy Cleveland Sullivan, whose claim to fame was having been hit by lightning on seven different occasions and surviving all of them. The details of these events are both tragic and a little hilarious, and raise some interesting questions about rationality.

From the article:

In spring 1972, Sullivan was working inside a ranger station in Shenandoah National Park when another strike occurred. It set his hair on fire; he tried to smother the flames with his jacket. He then rushed to the restroom, but couldn’t fit under the water tap and so used a wet towel instead. Although he never was a fearful man, after the fourth strike he began to believe that some force was trying to destroy him and he acquired a fear of death. For months, whenever he was caught in a storm while driving his truck, he would pull over and lie down on the front seat until the storm passed. He also began to believe that he would somehow attract lightning even if he stood in a crowd of people, and carried a can of water with him in case his hair was set on fire.

Put yourself in his situation and ask yourself if you might start doing the same thing after four times. Now what about if it kept happening?

On August 7, 1973, while he was out on patrol in the park, Sullivan saw a storm cloud forming and drove away quickly. But the cloud, he said later, seemed to be following him. When he finally thought he had outrun it, he decided it was safe to leave his truck. Soon after, he was struck by a lightning bolt.

The next strike, on June 5, 1976, injured his ankle. It was reported that he saw a cloud, thought that it was following him, tried to run away, but was struck anyway. His hair also caught fire.

He was struck the seventh time while fishing in a freshwater pool, which in a weird turn of events was followed by a confrontation with a bear over some trout that he had caught.

What’s more, Sullivan claimed to have been struck by lightning another time as a child, when out helping his father cut wheat in a field.

And furthermore…

Sullivan’s wife was also struck once, when a storm suddenly arrived as she was out hanging clothes in their backyard. Her husband was helping her at the time, but escaped unharmed.

Apparently his fear of lightning was a little contagious:

He was avoided by people later in life because of their fear of being hit by lightning, and this saddened him. He once recalled “For instance, I was walking with the Chief Ranger one day when lightning struck way off (in the distance). The Chief said, ‘I’ll see you later.'”

Okay, so besides from being a hilariously weird series of events, this article does raise some issues related to anthropic reasoning. Namely: what would it be rational for Roy Sullivan to believe?

I want to say that this man had really really good evidence that some angry Thor-like deity existed and was actively hunting him down. In his position, I think I’d feel like it was only rational to try to run from approaching clouds and thunderstorms (although that strategy didn’t seem to be super effective for him).

But at the same time, in a world of billions of people, it’s almost guaranteed to be the case that somebody will find themselves in circumstances just as unlikely as this. If Sullivan had one day looked up lightning strike statistics, and found that the numbers for the overall population were perfectly consistent with a naturalistic hypothesis in which lightning doesn’t target any particular individuals, how should he have responded?

And what should we believe about Roy Sullivan and lightning? Presumably we should not accept his non-naturalistic conclusions. But then what exactly is the difference between what we know and what he knows? We both have the same statistical information about the general trends in lightning strikes, and we both know that Roy Sullivan Cleveland was hit by lightning a bunch of times, so why should we come to different conclusions?

The obvious thought here is that it has something to do with anthropic reasoning. Sure, I have the same non-anthropic evidence as Roy Sullivan, but we have different anthropic evidence. Sullivan doesn’t just know the comparatively unremarkable proposition that “somebody was hit by lightning seven times and survived,” he knows the indexical proposition that “I was hit by lightning seven times and survived.” The non-Sullivans of the world don’t have access to this proposition, and maybe  this is the difference that matters.

Perhaps any population will end up having some individuals that happen to find themselves in very unusual situations, in which it becomes rational for them to come to bizarre conclusions about the world for anthropic reasons. And the bigger the population, the stranger and more rationally certain these beliefs might become.  Imagine a population big enough that it becomes not unlikely that some individual walks around commanding Thor to send bolts of lightning where they’re pointing, and then lo and behold it happens each time.

There would be many many many more individuals out there who succeeded a few times, and even more that never succeed at doing so. But for that tiny fraction that appears to manifest god-like powers, what should they believe? What should their friends and family believe? How far does the anthropic update extend? I’m not sure.

# A probability puzzle

To be totally clear: the question is not assuming that there is ONLY one student whose neighbors both flipped heads, just that there is AT LEAST one such student. You can imagine that the teacher first asks for all students whose neighbors both flipped heads to step forward, then randomly selected one of the students that had stepped forward.

It seemed initially obvious to me that the teacher was correct. There are exactly as many possible worlds in which the three students are HTH as there worlds in which they are HHH, right? Knowing how your neighbors’ coins landed shouldn’t give you any information about how your own coin landed, and to think otherwise seems akin to the Gambler’s fallacy.

But in fact, the teacher is wrong! It is in fact more likely that the student flipped tails than heads! Why? Let’s simplify the problem.

Suppose there are just three students standing in a circle (/triangle). There are eight possible ways that their coins might have landed, namely:

HHH
HHT
HTH
HTT
THH
THT
TTH
TTT

Now, the teacher asks all those students whose neighbors both have “H” to step forward, and AT LEAST ONE steps forward. What does this tell us about the possible world we’re in? Well, it rules out all of the worlds in which no student could be surrounded by both ‘H’, namely… TTT, TTH, THT, and HTT. We’re left with the following…

HHH
HHT
HTH
THH

One thing to notice is that we’re left with mostly worlds with lots of heads. The expected total of heads is 2.25, while the expected total of tails is just 0.75. So maybe we should expect that the student is actually more likely to have heads than tails!

But this is wrong. What we want to see is what proportion of those surrounded by heads are heads in each possible world.

HHH: 3/3 have H (100%)
HHT: 0/1 have H (0%)
HTH: 0/1 have H (0%)
THH: 0/1 have H (0%)

Since each of these worlds is equally likely, what we end up with is a 25% chance of 100% heads, and a 75% chance of 0% heads. In other words, our credence in the student having heads should be just 25%!

Now, what about for N students? I wrote a program that does a brute-force calculation of the final answer for any N, and here’s what you get:

 N cr(heads) ~ 3 1/4 0.25 4 3/7 0.4286 5 4/9 0.4444 6 13/32 0.4063 7 1213/2970 0.4084 8 6479/15260 0.4209 9 10763/25284 0.4246 10 998993/2329740 0.4257 11 24461/56580 0.4323 12 11567641/26580015 0.4352 13 1122812/2564595 0.4378 14 20767139/47153106 0.4404 15 114861079/259324065 0.4430 16 2557308958/5743282545 0.4453 17 70667521/157922688 0.4475

These numbers are not very pretty, though they appear to be gradually converging (I’d guess to 50%).

Can anybody see any patterns here? Or some simple intuitive way to arrive at these numbers?

# Solving the common knowledge puzzle

I’ve written up the way that I solved the puzzle, step by step:

1. A looks at B and C and says “I don’t know what my number is.”

We assess what information this gives us by considering in which scenario A would have known what their number was, and ruling it out.

Well, for starters, if A saw two different numbers, then A would consider there to be two logically possible worlds, one in which they are the sum of the two numbers, and another in which they are one of the two that are added together. But if A saw the SAME two numbers, they A would know that they must be the sum of those two (since zero is not a possible value for themselves). What this means is that the fact that A doesn’t know their number tells us with certainty that B ≠ C! Furthermore, since B and C will go through this same line of reasoning themselves, they will also know that B ≠ C. And since all of them know that B and C will go through the same line of reasoning, it becomes common knowledge that B ≠ C.

Good! So after A’s statement, we have added one piece of common knowledge, namely that B ≠ C.

2. B thinks a moment and then says “Me neither.”

Ok, one thing this tells us is that A ≠ C, for the exact same reason as before.

But it also tells us more than this, because B knows that B ≠ C and still doesn’t know. So we just need to think of the scenario in which knowing that B ≠ C (and knowing the values of A and C, of course) would tell B the value that they have. Try to figure it out for yourselves!

The answer is, the scenario is that A = 2C! Imagine that B sees that A is 10 and C is 5. This tells them that they are either 5 or 15. But they know they can’t be 5, because C is five and B ≠ C. So they must be 15! In other words, since they would know their value if A equaled 2C and they don’t know their value, this tells us that A ≠ 2C!

So now we have two more pieces of common knowledge: A ≠ C, and A ≠ 2C. Putting this together with what we knew before, we have a total of three pieces of information:

B ≠ C
A ≠ C
A ≠ 2C

3. C thinks a moment and then says “Me neither.”

By exact analogy with the previous arguments, this tells us that A ≠ B, as well as that A ≠ 2B and B ≠ 2A. (We saw previously that B could conclude from B ≠ C that A ≠ 2C. By the same arguments, C can conclude from C ≠ A that B ≠ 2A. And from C ≠ B, C can conclude that A ≠ 2B.)

There’s one more piece of information that we haven’t taken into account, which is that A ≠ 2C. In which situation does the fact that A ≠ 2C tell C their value? Well, if A = 10 and B = 15, then C is either 5 or 20. But C can’t be 5, because then A would be twice C. So C could conclude that they are 20. Since C doesn’t conclude this, we know that 3A ≠ 2B.

Putting it all together, we know the following:

B ≠ C
A ≠ C
A ≠ 2C
A ≠ 2B
B ≠ 2A
3A ≠ 2B

4. A thinks, and says: “Now I know! My number is 25.”

The question we need to ask ourselves is what the values of B and C must be in order that the above six conditions to allow A to figure out their own value. Pause to think about this before reading on…

Let’s work through how A processes each piece of information:

A could figure out their own value by seeing B = C. But we already know that this isn’t the case.

Since A knows that A ≠ C, A could figure out their value by seeing B = 2C. So that’s a possibility… Except that if B = 2C, then A = B + 2C = 3C. But 25 is not divisible by 3. So this can’t be what they saw.

Since A knows that A ≠ B, A could figure out their value by seeing C = 2B. But again, this doesn’t work, since it would imply that A was divisible by 3, which it is not.

Since A knows that A ≠ 2C, A could figure out their value by seeing B = 3C (e.g. B = 15, C = 5). They would rule out themselves being one component of the sum, and conclude that they are 4C. But 25 is not divisible by 4. So this is not the case.

Since A knows that A ≠ 2B, A could figure out their value by seeing C = 3B (e.g. B = 5, C = 15). By the same reasoning as before, this cannot be the case.

Since B ≠ 2A, A could figure out their value by seeing 3B = 2C (e.g. B = 10, C = 15). They would know that they cannot be just one component of the sum, so they would conclude that they must be B + C, or 2.5 B. Now, is there an integer B such that 25 = 2.5 B? You betcha! B = 10, and C = 15!

We can stop here, since we’ve found a logically consistent world in which A figures out that their own value is 25. Since there can only be one such world (as the problem statement implies that this information is enough to solve the puzzle), we know that this must be what they saw. So we’ve found the answer! (A,B,C) = (25,10,15). But if you’re curious, I’d suggest you go through the rest of the cases and show that no other values of B and C would be consistent with them knowing that their own value is 25.

One thing that’s interesting here is the big role that the number 25 played in this. The fact that 25 was not divisible by 3 but was divisible by 2.5, was crucial. For the same puzzle but a different value that 25, we would have come to a totally different answer!

My challenge to anybody that’s made it this far: Consider the set of all integers that A could have truthfully declared that they knew themselves to be. For some such integers, it won’t be the case that A’s declaration is sufficient for us to conclude what B and C are. Which integers are these?

# A common knowledge puzzle

Common knowledge puzzles are my favorite. Here’s one I just came across. I challenge you to try to figure it out in less than 5 minutes. 🙂

Three perfect logicians with positive (non-zero) integers taped to their foreheads, A, B, and C, sit in a triangle. Each doesn’t know their own number but can see the numbers for the other two. It is common knowledge amongst all three that one of the numbers is the sum of the other two, but it is not known which is which.

A looks at B and C, and says “I don’t know what my number is.”

B thinks a moment and then says “Me neither.”

C thinks a moment and then says “Me neither.”

A thinks, then says “Now I know! My number is 25.”

What are the numbers on B and C’s foreheads?

(Solution here)