Social Experiments: Bad Criticism, Good Criticism

A New York Times article everybody and their brother link to discusses a social experiment conducted on behalf of the city of New York:

It has long been the standard practice in medical testing: Give drug treatment to one group while another, the control group, goes without.

Now, New York City is applying the same methodology to assess one of its programs to prevent homelessness. Half of the test subjects — people who are behind on rent and in danger of being evicted — are being denied assistance from the program for two years, with researchers tracking them to see if they end up homeless.

Sounds reasonable? Not everybody thinks so. Here's one critic:

“They should immediately stop this experiment,” said the Manhattan borough president, Scott M. Stringer. “The city shouldn’t be making guinea pigs out of its most vulnerable.”

Here's another:

“I don’t think homeless people in our time, or in any time, should be treated like lab rats,” Ms. Palma said.

Oh boy! If the New York Times had wanted a critical perspective that's a little more, um, high-brow, they might have asked Robert J. Sampson. Here's a longish excerpt from his recent paper "Gold Standard Myths: Observations on the Experimental Turn in Quantitative Criminology" (pp. 492-93):

Possibly the most inelegant phrase to have entered the social science lexicon, the ‘‘stable unit treatment value assumption’’ (SUTVA) underlies all experiments. Although not necessarily of great consequence for research on fertilizers or possibly taking a pill, SUTVA is foundational to criminology because at bottom it refers to assumptions about social interactions and what is known in the statistical literature as ‘‘interference’’ (Rosenbaum 2007; Sobel 2006). The essential idea of SUTVA is that potential outcomes for any unit of an experiment are independent of the treatment assignment of any other unit or population member under study. Human response to treatments like ‘‘Hawthorne’’ or ‘‘John Henry’’ effects (when participants in the control group alter their behavior purposely because of the experiment) are thus ruled out, as, more problematically, are any forms of social interaction. I quote here a summary in words of a very dense, technical literature:

If, through processes of information diffusion, norm formation, leadership, endogenous reinforcement, or competition in tournaments, social interactions are important for outcomes, units effects are inherently undefined because, in this case, outcomes for any particular unit i depend on the number or distribution of treated units j in the population, and the standard interpretation of any parameter estimate no longer applies (Gangl 2010, p. 40).

The implications of SUTVA are profound. Processes of contagion, information diffusion, jealousy, and learning are staples of criminology and the social sciences in general. Raudenbush (2008), for example, notes how the very nature of educational processes leads to violations of SUTVA, calling into question the experimental turn in that discipline as well. Yet with few exceptions the experimental literature in criminology, and apparently education, has not tackled interference in a straightforward way. Consider just a few examples of the pervasiveness of the problem for criminology. Berk (2005) notes that placing a substantial number of rival gang members in the same boot camp could dramatically alter the nature of the treatment and the subsequent response (p. 421). In another, "suppose a drug treatment program was randomly provided to some youth facilities and not others. The potential benefits of the drug treatment program for individuals could be enhanced or reduced because of peer pressure among juveniles, all of whom were participating in the same program (p. 421)."

He goes on to discuss the famous MTO housing experiment.

Neither Sampson nor I want to do away with social experiments, but the regrettable fact is that people tend to influence one another, which makes the interpretation of results harder than it otherwise would be. I don't agree with everything in the strident paper - in particular, it culminates in what amounts to a Feyerabendian Anything Goes* - but it is short, available in full for free, and one of the few papers ever to be published in the Journal of Quantitative Criminology that are entirely mathsless. Recommended weekend reading for anyone for whom the New York Times is not quite good enough. That is, pretty much every reader of this blog. Boy, what a clever bunch we are!

______
*I don't really care about social experiments. The real reason I'm writing this is that I was looking for an excuse to use the phrase "Feyerabendian Anything Goes". Mission accomplished!