After All, It's Other People's Money

Contains profanity

It's a point I've made before, but it's worth repeating.

jdc325 reviews Ian Ayres' Supercrunchers. An excerpt:

The book also looks at testing social policy (I wrote about recidivism and sentencing recently) in the US and there is also work being done on testing social policy in the UK. There is a link at the bottom of this page to a paper about RCTs in social intervention in the UK and in 2007 Ben Goldacre wrote about a Professor and statistician named Sheila Bird who wished to do controlled trials on social policy.

Do click on the links.

Usually, jdc blogs about "alternative medicine" and the nonuse of the scientific method in medical contexts. That is certainly a worthwhile topic, but people who focus on that may not realize how relatively good things are in the world of medicine: It is generally accepted that a new medicine should be tested by using randomized controlled trials. In contrast, the idea that policies which fall into the realm of the social sciences should be tested in a similar fashion is wildly exotic. To cite numbers from a few years ago, there were about 500 drug use prevention programmes in Germany, none of which had ever been evaluated.

Counter-argument 1: You can't test the efficacy of such programmes like you can test the efficacy of new medical treatments. Answer: Bollocks. Assign (preferrably nonadjacent) regions to a treatment or control condition on a random basis and see what happens.

Counter-argument 2: Evaluations cost money. Answer: True. But drug use prevention programmes cost money, too, and we may well be wasting money on programmes that don't work or are even harmful. As for the subgroup of programmes that do work, I'd like to know how to get the biggest bang for the buck. Evaluations would have to be incredibly expensive or differences between programmes implausibly small for such a course of action not to be cost-effective.

Whenever possible, new policies should be evaluated in a rigorous manner. The fact that this does not happen is an ongoing scandal. It's just that nobody notices because things have always been that way.