A story about Bayes, Part 1: Binary search

When I was 19 and just beginning my PhD, I found myself with a lot of free time and flexibility in my schedule. Naturally, I decided to figure out which dietary supplements I should take.

Phase 1: The design

First of all, on priors, I expected a fair amount of individual variation in responses to diet, supplements, drugs, and so on. I knew that two different people eating the same food and doing the same exercises could be overweight and underweight, respectively. And I didn’t just want to know which supplements were good “for people on average”. I wanted to know what was good for me.

I reasoned that I could take various blood tests, and would eventually do that. But first, there was an experiment I needed to do. One that would be confounded by knowing the results of a blood test. One that I needed to do on my own. I wanted to see what effects I could personally observe from taking supplements, with minimal confounding from placebo effects that would be caused by expectations about what I needed — expectations that a blood test would lead to.

There were a lot of supplements, and it would take a long time to go through all of them, one by one. Luckily, a mathematical trick came to mind, called binary search, whereby under certain conditions you can search through a list of n things in log_2(n) time. Pretty nifty.

So, I decided to take all the supplements. Okay, not all of them. I stuck to a list of commonly known ones, and did a bit of research first to see if there were any serious interactions between them, while steering clear of discussions about their “typical effects”: a multivitamin, vitamins A, B, C, D, and E, omega 3, 6, and 9, zinc, calcium, magnesium, potassium, and some digestive enzymes: 14 supplements. In particular, fewer than 16.

I do not recommend doing this. Subsequent experience seeing folks experimenting with supplements have made me more cautious. Nonetheless, the story remains, and I’m going to tell it now.

Phase 2: The search

When I started taking my batch of 14 supplements each day, within a couple of days, I felt.. better. More energy. Nice! I laughed more, and got more work done. Not bad for a placebo effect, if that’s all that was happening.

Next, I flipped a coin, which said to eliminate the first 7 supplements on my list, so I’d only be taking supplements 8-14. After a couple of days, I felt worse again. For redundancy (as this step is not required in an ideal binary search), I switched to taking 1-7, and felt better again. Interesting. Then I switched by coin flip to taking 4-7, still felt good, switched to 6-7, felt worse, switched to 4-5, felt better, and finally switched to 5 — a vitamin C supplement — and continued to feel great. In particular, this mostly ruled out the possibility that I needed to take a combination of supplements to feel great, which had been a possibility at the outset that would have been trickier to find via binary search.

Anyway, having done this, there were two main possibilities: either the vitamin C supplements were particularly observably good for me, which would explain the data, or they weren’t, and this just happened randomly.

So I applied some probability theory. I assigned a reasonably high prior probability that at least one of those supplements would be good for me — say, 70% — and so a prior of at least 5% to any particular one of the 14 being good. In particular, I assigned a prior on the order of 1:20 that the vitamin C supplement was actually good for me.

Now let X be the random variable indicating whatever supplement my binary search would end with, and let Y be the statement “The vitamin C supplement is the one I need most”. I reasoned that P(X = vitamin C | Y) was higher than P(X = vitamin C| not Y). So I needed to increase that 1:20. How much? I figured maybe there could be several vitamins I’d benefit from, on priors, so maybe P(X = vitamin C | Y) was around 50% or something, vs P(X = vitamin C | not Y) — the probability of settling on the vitamin C supplement randomly — being around 1/14 = 7%. Thus, I figured I should make around a 50%:7% = 7:1 update in favor of taking the C supplement, taking my prior to from 1:20 to around 7:20, or 1:3, for taking vitamin C.

Since vitamin C was fairly harmless, I decided 1:3 was credence enough to keep taking it.

Phase 3: After the experiment

I continued to feel particularly good, except on various days when I forgot to take it, giving me what I considered a 3:1 update from 1:3 to 1:1, or 50%, that the vitamin C supplement was having the observed effect.

That update would have been larger if not for the possibility that, at this point in the story, I started having some hopes and expectations particularly around vitamin C, and so the placebo effect could explain the continued effect. In other words, another, more complex hypothesis could explain the data: it was possible that the days on which I felt good were delayed reactions to supplements I felt on prior days, and the reason I continued to feel good after the experiment, having discontinued those, was a placebo effect coming from my expectation that the experiment had worked. However, I reasoned that, for that to have happened, I would have needed a good-for-me supplement on each day that was two days prior to feeling good during the study, which would have been an unlikely coincidence. In other words, the “Vitamin C supplement is good for me” hypothesis explained the observations much better than the “delayed effects + placebo effects” hypothesis, which nonetheless occupied a good chunk the remaining 50% of my credence.

Anyway, I kept taking the C supplement.

Phase 4: A surprise

Then one day, I had to travel, and I couldn’t take my vitamin C powder on the plane, so I took tablets… and I felt worse. Maybe just due to the travel? Perhaps, but when I got home, alternating between tablets and powder still made be feel worse on tablet days. Huh. In particular, this data was evidence against the placebo theory: that expectations about vitamin C were sufficient to produce my observations. So I made a 2:1 update from 1:1 to 2:1 in favor of the vitamin C powder being responsible for the effect.

Moreover, I remarked that while I had figured out that this particular vitamin C supplement was reasonably likely (2:1) to be good for me, I had never really tested whether vitamin C itself was the reason. It might have been a placebo effect I derived from powders, or some other ingredient. I hadn’t prioritized reading all the ingredients because all I cared about was whether I should take this supplement? and not why I should take it.

And as it turned out, the powder also contained a mixture of B vitamins. Huh! So I switched to taking a vitamin B complex tablet, and lo, did not feel worse… evidence in favor of the other-ingredients hypothesis, explaining away the placebo-powder hypothesis, giving me another 2:1 update, from 2:1 to 4:1, in favor of the effect being real and non-placebo. Thus I now had a 4:1 credence — around 80% — that “something in that vitamin C powder and/or these tablets is good for me”.

Now, I had only delayed getting a blood test to avoid creating expectations that I needed any particular supplement. Since I was now developing some pretty clear expectations, there was no point in delaying the test further. And lo, the blood test found that I was vitamin B deficient. It was hard to find research on this, but I figured the probability of being B vitamin deficient given that B vitamins are a good thing for me to take was at least three times as high as if they weren’t a good thing to take, giving me a 3:1 update from 4:1 to 12:1 in favor of taking the supplement.

So, I kept taking it.

And, with the experiments over, I did a bunch more reading about particular B vitamins and why I might need them, but I’ll spare those details here. They’re not the point of the story.

The real point of the story comes 10 years later…