That's Not the Fast Lane to Stockholm [Edited]

You've probably heard it said that absence of evidence is not evidence of absence. Elizer Yudkowsky adds nuance:

Absence of evidence is evidence of absence only to the degree that we could reasonably expect the evidence to appear. If someone is trumpeting that snake oil cures cancer, you can reasonably expect that, if the snake oil was actually curing cancer, some scientist would be performing a controlled study to verify it - that, at the least, doctors would be reporting case studies of amazing recoveries - and so the absence of this evidence is strong evidence of absence. But "gaps in the fossil record" are not strong evidence against evolution; fossils form only rarely, and even if an intermediate species did in fact exist, you cannot expect with high probability that Nature will obligingly fossilize it and that the fossil will be discovered.

[...]

The general mistake probably arises because there are cases where the absence of scientific proof is strong evidence - because an experiment would be readily performable, and so failure to perform it is itself suspicious. (Though not as suspicious as I used to think - with all the strangely varied anecdotal evidence coming in from respected sources, why the hell isn't anyone testing Seth Roberts's theory of appetite suppression?)

That's a good question. Assuming that by "testing", Elizer means a randomized controlled trial, I think there are a number of factors relevant here. I'll start with those that should make the existence of a test more likely:

1. It's an important topic that presumably lots of people do research on.

2. It's a topic high on the political agenda in many western countries. (Translation: Lots of research funding available)

3. The theory makes sense.

4. The theory is in line with previous evidence, which Roberts cites.

5. As noted above, there is anecdotal evidence to support it.

6. Testing the theory - i.e., the diet which he devised on the basis of it* - would be relatively cheap. I have not done a power analysis on this one, but from the top of my head, I would guess that if you have an experimental and a control group of 150 each, that should be enough to find significant effects after a year or so - if it works as well as Roberts says it does. Given that many people want to lose weight, you may have to pay your subjects very little.

But:

7. On the other hand, the diet is sort of messy - there are different ways of doing it. If you want to test all of the ways, your number of subjects is going to grow - a lot. I earlier proposed that one way to test the diet would be just to tell people about the different methods and let them try out what works best for them; this would be pretty unconventional, I believe.

8. The theory was not published in a peer-reviewed journal, or indeed any journal. (I don't know why.)

9. Roberts himself has stressed that he's an outsider in the field, but a new look at his "About the Author" page gave me second thoughts in this respect. It lists quite a few papers of his with relevance to nutrition. Also, it says that he serves on the editorial board of the journal Nutrition. (I think that's rather recent, but I'm not sure.) That's not an outsider in my book.

10. However, it also informs us he is now an emeritus. Presumably, that reduces one's clout.

11. The most important point: The psychologist David Lykken - and probably others before and after him - pointed out that in the scientific marketplace it isn't good enough to put a sound idea out there and hope that other people will start to work on it. You have to push it - promote it, if you will. A successful large randomized controlled trial would presumably go a long way in convincing the scientific community of the theory's accuracy - Roberts, however, has explicitly said he's against such a trial at the present moment (e.g., see his comments at this post). [In the comments section, Roberts points out that "[p]sychologists never do large randomized studies" and that he promotes the idea in other ways. That's true, but a relatively large randomized trial would be "a big convincer", as Andrew Gelman put it in the post linked to.]

On balance, maybe we shouldn't be too surprised.

*For those not in the know, a very short description of the Shangri-La Diet (rehashed from an earlier post):

The theory behind the diet is based on the notion that there is a body-weight set point. The set point is what the brain wants the body to weigh. The mechanism with which the body signals this is hunger (or lack thereof). To illustrate, if you are ten kilos below your set point, you'll be hungry pretty much all of the time.

Roberts argues that the set-point has a default tendency to slowly decrease over time. However, it will rise if you experience taste sensations that your brain associates with calories - if you eat familiar-tasting food.

The idea behind the Shangri-La diet is to lower the set point by giving the body calories from food that does not come with familiar tastes. Methods include: 1. Drinking extra light (!) olive oil (which is nearly tasteless) between meals. The between-meals bit is important so that the oil's calories won't get associated with the taste of the meals. For the same reason, the consumption of the oil must not go with any other tastes (cigarettes, gum, etc.). 2. "Crazy spicing": When you have a meal, use a random selection of spices to flavour it, so that the taste of the food is unusual every time. (This doesn't sound particularly attractive to me.) 3. Nose-clipping during meals. This reduces the taste sensation and thus the flavour-calorie association.