Social Experiments: Bad Criticism, Good Criticism

A New York Times article everybody and their brother link to discusses a social experiment conducted on behalf of the city of New York:
It has long been the standard practice in medical testing: Give drug treatment to one group while another, the control group, goes without.

Now, New York City is applying the same methodology to assess one of its programs to prevent homelessness. Half of the test subjects — people who are behind on rent and in danger of being evicted — are being denied assistance from the program for two years, with researchers tracking them to see if they end up homeless.
Sounds reasonable? Not everybody thinks so. Here's one critic:
“They should immediately stop this experiment,” said the Manhattan borough president, Scott M. Stringer. “The city shouldn’t be making guinea pigs out of its most vulnerable.”
Here's another:
“I don’t think homeless people in our time, or in any time, should be treated like lab rats,” Ms. Palma said.
Oh boy! If the New York Times had wanted a critical perspective that's a little more, um, high-brow, they might have asked Robert J. Sampson. Here's a longish excerpt from his recent paper "Gold Standard Myths: Observations on the Experimental Turn in Quantitative Criminology" (pp. 492-93):
Possibly the most inelegant phrase to have entered the social science lexicon, the ‘‘stable unit treatment value assumption’’ (SUTVA) underlies all experiments. Although not necessarily of great consequence for research on fertilizers or possibly taking a pill, SUTVA is foundational to criminology because at bottom it refers to assumptions about social interactions and what is known in the statistical literature as ‘‘interference’’ (Rosenbaum 2007; Sobel 2006). The essential idea of SUTVA is that potential outcomes for any unit of an experiment are independent of the treatment assignment of any other unit or population member under study. Human response to treatments like ‘‘Hawthorne’’ or ‘‘John Henry’’ effects (when participants in the control group alter their behavior purposely because of the experiment) are thus ruled out, as, more problematically, are any forms of social interaction. I quote here a summary in words of a very dense, technical literature:
If, through processes of information diffusion, norm formation, leadership, endogenous reinforcement, or competition in tournaments, social interactions are important for outcomes, units effects are inherently undefined because, in this case, outcomes for any particular unit i depend on the number or distribution of treated units j in the population, and the standard interpretation of any parameter estimate no longer applies (Gangl 2010, p. 40).
The implications of SUTVA are profound. Processes of contagion, information diffusion, jealousy, and learning are staples of criminology and the social sciences in general. Raudenbush (2008), for example, notes how the very nature of educational processes leads to violations of SUTVA, calling into question the experimental turn in that discipline as well. Yet with few exceptions the experimental literature in criminology, and apparently education, has not tackled interference in a straightforward way. Consider just a few examples of the pervasiveness of the problem for criminology. Berk (2005) notes that placing a substantial number of rival gang members in the same boot camp could dramatically alter the nature of the treatment and the subsequent response (p. 421). In another, "suppose a drug treatment program was randomly provided to some youth facilities and not others. The potential benefits of the drug treatment program for individuals could be enhanced or reduced because of peer pressure among juveniles, all of whom were participating in the same program (p. 421)."
He goes on to discuss the famous MTO housing experiment.

Neither Sampson nor I want to do away with social experiments, but the regrettable fact is that people tend to influence one another, which makes the interpretation of results harder than it otherwise would be. I don't agree with everything in the strident paper - in particular, it culminates in what amounts to a Feyerabendian Anything Goes* - but it is short, available in full for free, and one of the few papers ever to be published in the Journal of Quantitative Criminology that are entirely mathsless. Recommended weekend reading for anyone for whom the New York Times is not quite good enough. That is, pretty much every reader of this blog. Boy, what a clever bunch we are!

*I don't really care about social experiments. The real reason I'm writing this is that I was looking for an excuse to use the phrase "Feyerabendian Anything Goes". Mission accomplished!


Steve Sailer said...


For example, in the late 1960s it seemed cool to be a criminal, and for the last 10 or 15 years, it has seemed uncool and stupid.

But how do you do a Jim Manzi-style double blind test on something like that, which depends on the mass media, older siblings' attitudes, and endless other factors? It would be nice to be able to do experiments like that, but I don't know how to experiment on the Spirit of the Age.

LemmusLemmus said...

The examples you mention illustrate the fact that there are lots of factors that don't lend themselves easily to manipulation by experimenters. They also illustrate that social life contains convoluted back-and-forth influences between causal factors, including outcome-to-outcome influences. No wonder sociologists are envious of physicists!

But in the excerpt I quoted, Sampson correctly points out that even if you can manipulate the variable of interest and randomly assign individuals to one or the other condition, the interpretation of the results is not straightforward because one subject's response is "contaminated" by other subjects' responses. In other words, social experiments are problematic even under favourable conditions.