15/05/2008

That's Not the Fast Lane to Stockholm [Edited]

You've probably heard it said that absence of evidence is not evidence of absence. Elizer Yudkowsky adds nuance:

Absence of evidence is evidence of absence only to the degree that we could reasonably expect the evidence to appear. If someone is trumpeting that snake oil cures cancer, you can reasonably expect that, if the snake oil was actually curing cancer, some scientist would be performing a controlled study to verify it - that, at the least, doctors would be reporting case studies of amazing recoveries - and so the absence of this evidence is strong evidence of absence. But "gaps in the fossil record" are not strong evidence against evolution; fossils form only rarely, and even if an intermediate species did in fact exist, you cannot expect with high probability that Nature will obligingly fossilize it and that the fossil will be discovered.


[...]

The general mistake probably arises because there are cases where the absence of scientific proof is strong evidence - because an experiment would be readily performable, and so failure to perform it is itself suspicious. (Though not as suspicious as I used to think - with all the strangely varied anecdotal evidence coming in from respected sources, why the hell isn't anyone testing Seth Roberts's theory of appetite suppression?)

That's a good question. Assuming that by "testing", Elizer means a randomized controlled trial, I think there are a number of factors relevant here. I'll start with those that should make the existence of a test more likely:

1. It's an important topic that presumably lots of people do research on.

2. It's a topic high on the political agenda in many western countries. (Translation: Lots of research funding available)

3. The theory makes sense.

4. The theory is in line with previous evidence, which Roberts cites.

5. As noted above, there is anecdotal evidence to support it.

6. Testing the theory - i.e., the diet which he devised on the basis of it* - would be relatively cheap. I have not done a power analysis on this one, but from the top of my head, I would guess that if you have an experimental and a control group of 150 each, that should be enough to find significant effects after a year or so - if it works as well as Roberts says it does. Given that many people want to lose weight, you may have to pay your subjects very little.

But:

7. On the other hand, the diet is sort of messy - there are different ways of doing it. If you want to test all of the ways, your number of subjects is going to grow - a lot. I earlier proposed that one way to test the diet would be just to tell people about the different methods and let them try out what works best for them; this would be pretty unconventional, I believe.

8. The theory was not published in a peer-reviewed journal, or indeed any journal. (I don't know why.)

9. Roberts himself has stressed that he's an outsider in the field, but a new look at his "About the Author" page gave me second thoughts in this respect. It lists quite a few papers of his with relevance to nutrition. Also, it says that he serves on the editorial board of the journal Nutrition. (I think that's rather recent, but I'm not sure.) That's not an outsider in my book.

10. However, it also informs us he is now an emeritus. Presumably, that reduces one's clout.

11. The most important point: The psychologist David Lykken - and probably others before and after him - pointed out that in the scientific marketplace it isn't good enough to put a sound idea out there and hope that other people will start to work on it. You have to push it - promote it, if you will. A successful large randomized controlled trial would presumably go a long way in convincing the scientific community of the theory's accuracy - Roberts, however, has explicitly said he's against such a trial at the present moment (e.g., see his comments at this post). [In the comments section, Roberts points out that "[p]sychologists never do large randomized studies" and that he promotes the idea in other ways. That's true, but a relatively large randomized trial would be "a big convincer", as Andrew Gelman put it in the post linked to.]

On balance, maybe we shouldn't be too surprised.

*For those not in the know, a very short description of the Shangri-La Diet (rehashed from an earlier post):

The theory behind the diet is based on the notion that there is a body-weight set point. The set point is what the brain wants the body to weigh. The mechanism with which the body signals this is hunger (or lack thereof). To illustrate, if you are ten kilos below your set point, you'll be hungry pretty much all of the time.

Roberts argues that the set-point has a default tendency to slowly decrease over time. However, it will rise if you experience taste sensations that your brain associates with calories - if you eat familiar-tasting food.

The idea behind the Shangri-La diet is to lower the set point by giving the body calories from food that does not come with familiar tastes. Methods include: 1. Drinking extra light (!) olive oil (which is nearly tasteless) between meals. The between-meals bit is important so that the oil's calories won't get associated with the taste of the meals. For the same reason, the consumption of the oil must not go with any other tastes (cigarettes, gum, etc.). 2. "Crazy spicing": When you have a meal, use a random selection of spices to flavour it, so that the taste of the food is unusual every time. (This doesn't sound particularly attractive to me.) 3. Nose-clipping during meals. This reduces the taste sensation and thus the flavour-calorie association.

9 comments:

Seth Roberts said...

"Roberts, on the other hand, actively opposes testing the theory/the diet based on it at the present moment." You're wrong. I am now helping others do such tests.

If you read again the post you cite, you will see that I support small tests over large ones. Maybe that was confusing. Did anything else give you the idea that I opposed testing my theory?

LemmusLemmus said...

Well, I wrote above

'Assuming that by "testing", Elizer means a randomized controlled trial...'

but I could see how this is still misleading. Sorry, I didn't mean to misrepresent your views. I've changed it to:

'Roberts, on the other hand, actively opposes testing the theory/the diet based on it in a large randomized controlled trial at the present moment'

Is that accurate?

Seth Roberts said...

Thanks. It's still inaccurate -- I don't "actively" oppose such trials, since if someone decided to do one I would certainly not get in their way. More importantly it's highly misleading since I am actively supporting small randomized trials. Nobody could possibly figure that out from what you wrote.

Seth Roberts said...

And since small randomized trials are a step in the direction of large randomized trials, you could say I'm actively supporting large randomized trials, "supporting" in the sense of doing things that will make them more likely.

LemmusLemmus said...

In my view, saying one is against something qualifies as "actively opposing". But I've changed it again to: "Roberts, on the other hand, has explicitly said he's against testing the theory/the diet based on it in a large randomized controlled trial at the present moment".

Seth Roberts said...

Your changes haven't fixed the problem. Your point #11 says "It is important to promote an idea. Roberts, on the other hand . . . " -- as if I'm not promoting it! But I am. As I've said here twice, I am helping others do research that tests my ideas. David Lykken was a psychologist. Psychologists never do large randomized studies. Yet they promote their ideas all the time. There are many ways to promote scientific ideas, and they include what I am doing.

LemmusLemmus said...

The post didn't explicitly say that you didn't promote the idea (which would be ridiculous), but it could have been read this way. I changed the post again, this time to:

"You have to push it - promote it, if you will. A successful large randomized controlled trial would presumably go a long way in convincing the scientific community of the theory's accuracy - Roberts, however, has explicitly said he's against such a trial at the present moment (e.g., see his comments at this post). [In the comments section, Roberts points out that "[p]sychologists never do large randomized studies" and that he promotes the idea in other ways. That's true, but a relatively large randomized trial would be "a big convincer", as Andrew Gelman put it in the post linked to.]"

Which actually expresses what I meant to say better than previous versions. Please let me know if this is o.k.

I am under the impression that I made you somewhat angry. That certainly wasn't my intention!

Seth Roberts said...

well, at least it no longer says "Roberts . . . actively opposes testing the theory/the diet based on it" -- which makes me look like a nut.

"Somewhat angry"? Uh, yes, it is irritating to be portrayed as a nut and then, when you try to fix the mistake, it isn't fixed.

LemmusLemmus said...

Seth,

I think both of your points are answered above.