Biological Reality of Race? What Does It Even Mean? (Also: Free access to Sage journals)

Via Dan Hirschman at Scatterplot comes a debate in Sociological Theory about the nature of race: is it social and/or biological? The new contributions consist of three critical reactions to an article in the 2012 volume of the same journal by Jiannbin Lee Shiao, Thomas Bode, Amber Beyer and Daniel Selvig called "The Genomic Challenge to the Social Construction of Race", and a rejoinder by Shiao. 

The topic isn't new, and the sub-exchange between Shiao in one corner and Daniel Martinez HoSang in the other confirms what something I've long been thinking about this.  

As you may know, variants of cluster analysis can be used to group individuals' genomes on the basis of similarities and dissimilarities, and it has been shown that the resulting clusters correspond to racial categories, as measured by self-identification, for example. One of the two main arguments in the initial Shiao et al. paper is that this clearly shows that the view that race has no biological basis, held by so many sociologists, is wrong.

HoSang's article ends in an attempt at character assassination that stops just short of holding Shiao et al. personally responsible for the gas chambers in Auschwitz, but the earlier portions actually have serious content. HoSang voices misgivings about the validity of the cluster analyses and their interpretation by Shiao et al. and others, but then goes on to say (p. 233):

And even if one accepts the (contested) finding that self-identified race or ethnicity correlates with population structure, this finding does not justify a conclusion that “race” (or clinal class) has a biological basis. At the most quotidian level, the findings suggest that a statistical analysis of genetic ancestry informative markers of a population in the United States that self-identifies as “black” is likely to bear a relationship to an analysis of populations sampled in some region of sub-Saharan Africa. And a population that self-identifies as Chinese is likely to be statistically related with a population in China (Dupré 2008). That a new statistical technique has validated a high probability of such histories of migration is hardly revelatory; it does not establish a biological basis of race.

But  Shiao et al. clearly think just that: These findings show that race has a biological basis.

I suggest that people who wish to have this debate take a step back and start by reaching an agreement on the following:

1. What does it means to say, "Race has a biological basis"? What does it mean to say "Race is a biologically meaningful concept"? Are the two the same?

2. What evidence, if it existed, would show that race has a biological basis/is a biologically meaningful concept? What evidence, if it existed, would refute those claims?

If you don't do that, you'll debate ad infinitum.

Added: Along similar lines, Fabio Rojas comments.

**************

By the way, you can download all of the articles above, as Sage allows open access to all of its journals until October 31st (registration required).

Seth Roberts Is Dead

Today, from his siter Amy, via his blog, came the message that Seth Roberts has passed away. My condolences to his family and friends.

I never met him, only had a few exchanges with him on this and his blog. Generally, I felt he went to far in his criticism of standard approaches, and put too much weight on low-quality evidence. But, as long as I knew of his work - and I certainly include his blogging here - I valued him as an original, unusual, and stimulating thinker. I believe that once the great weight gain in affluent countries ca. 1970-present is better understood, his learning theory of the set point will be a large part of the explanation.

Here are posts in which I discuss Seths work (some of them quite critical):

• "Seth Roberts coins a new term"
• "The Best of Both Worlds"
•  "That's Not the Fast Lane to Stockholm"
• "Problems with Evidence-based Medicine"
• "The Long and Winding Post"
• "Original Medical Opinion of the Day"
• "Effect Size=Holy Crap!"
• "Do Rising Gas Prices Decrease Obesity?"
• "Appreciative Thinking and Cross-sectional Studies"

Two of his posts made it onto my year-end "Best Blogposts of..." lists:

• Will Like vs. Might Love vs. Might Hate
• Two Dimensions of Economic Growth: GDP and Useful Knowledge

Here are quotes of his that I found worth keeping. Here is his paper on self-experimentation in Behavioral and Brain Sciences. Here is his paper "What Makes Food Fattening?" Blowhard, Esq. remembers. Ben Casnocha remembers. Andrew Gelman remembers.

Around the Blogs, Vol. 106

1. The importance of being able to enjoy the mundane (Ben Casnocha)

2. "Nonshared environment" might best be conceived of as noise, not environment, says Kevin Mitchell.

3. External validity alert: Are patients in medical trials selected for large treatment effects? (Andrew Gelman/Paul Alper)

4. Chris Bertram makes a surprisingly good case for the argument "Squeezing the rich is good: even when it raises no money".

5. "Is there no racial bias precisely because it seems like there is?" Ole Rogeberg takes us into the mind of the microeconomist.

6. The fun of being criticized for doing the right thing, namely keeping the normative and the positive apart (Bryan Caplan)

7. An argument against men attending childbirth (Dennis Mangan) (via)

8. 50 great book covers from 2013, collected by Dan Wagstaff (via)

9. The low-hanging fruit of immigration: Bryan Caplan offers another metaphor.

10. The argument that extremely high cognitive talent is specific. (Bruce Charlton)

11. The argument for being an asshole (Scott Adams)

12. What's it like to hear voices that aren't there? (Christian Jarrett/L. Holt and A. Tickle)

13. Which works did sociologists cite most in 2013? (Neal Caren)

The Law of Yawn

Regular readers may remember a post about identification of causal effects I wrote in August. Here's the full text:

The better a model is at identifying a causal effect, the less likely it is the effect is going to look substantial. That's because of (i) publication bias, (ii) how the world works.

You may note that that text contains zero examples - it's just a general impression plus some armchair theorizing. Thankfully, Steve Sailer provides an example from commercial marketing research:

In fact, one side effect of bad quantitative methodologies is that they generate phantom churn, which keeps customers interested. For instance, the marketing research company I worked for made two massive breakthroughs in the 1980s to dramatically more accurate methodologies in the consumer packaged goods sector. Before we put to use checkout scanner data, market research companies were reporting a lot of Kentucky windage. In contrast, we reported actual sales in vast detail. Clients were wildly excited ... for a few years. And then they got kind of bored.

You see, our competitors had previously reported all sorts of exciting stuff to clients: For example, back in the 1970s they'd say: of the two new commercials you are considering, our proprietary methodology demonstrates that Commercial A will increase sales by 30% while Commercial B will decrease sales by 20%.

Wow.

We'd report in the 1980s: In a one year test of identically matched panels of 5,000 households in Eau Claire and Pittsfield, neither new commercial A nor B was associated with a statistically significant increase in sales of Charmin versus the matched control group that saw the same old Mr. Whipple commercial you've been showing for five years. If you don't believe us, we'll send you all the data tapes and you can look for yourselves.

Ho-hum.

In the social sciences - and I would include marketing - there probably are few cases when the effect of X on Y is genuinely zero. Just about everything influences everything else, in a roundabout way. There's a flipside to that: The influence of single factors is usually very small. A core reason for that is that people's personalities and behaviour are pretty stable, which is why the concept "personality" makes sense.

Of course, there's also Xs that have a large influence on Y. The problem is that researching this is, or soon becomes, pretty boring. In fact, when people say "Did we really need a study for that?", they sometimes have a point. When an influence is large, it will usually (though not by logical necessity) be readily apparent. You don't need to be a social scientist to see that adolescent's friends influence their behaviour.

So, shut up shop? I think not. One, you do need a social scientist to tell you how large an obvious effect is. Two, the above allows for a sweet spot where effects are not obvious, but large enough to detect. Three, and this is perhaps the most important point, it is a worthwhile endeavour to show that the effect of X on Y really is close to zero, contrary to what some people would have you believe. Especially if X costs money.

Around the Blogs, Vol. 103

Bit late today, but here's some recent posts that may be worth your time.

1. Andrew Gelman knows how randomization works in animal studies. (Post starts off with disturbing image)

2. Gabriel Rossman has tips on how to be a better journal reviewer, with a focus on decreasing turnaround times. Fabio Rojas links and summarizes.

3. Christian Jarrett summarizes a new paper by Brian D. Earp, Jim A. C. Everett, Elizabeth N. Madva, and J. Kiley Hamlin, who cannot replicate the "Macbeth effect", i.e., the finding that feelings of disgust increase the desire for physical cleaning.

New Paper: At Least One Method for Estimating the Effect of Genes Yields Misleading Results

Until recently, behavioural geneticists had to use twin samples to estimate the heritability of traits. The standard method estimates heritability - the contribution of genes - by exploiting the fact that identical twins are more genetically alike than nonidentical twins, who are more alike than unrelated people. A drawback of this method is that twin samples are hard to find. Recently, a method has become available that circumvents this problem; it's called genome-wide complex trait analysis (GCTA). The basic idea is to use differences in the actual genetic makeup of people to calculate heritability. It should not be confused with the genome-wide association technique, which "hunts" for specific "genes for" some outcome (the "gene for depression" or what have you).

A recent paper by Maciej Trzaskowski, Philip S. Dale and Robert Plomin (abstract; via) compares heritability estimates on the basis of standard and GCTA techniques. Results for the dependent variables show that GCTA estimates are considerably smaller than standard estimates for height, weight and intelligence. The real shocker are the results for "behaviour problems" (such as depression or hyperactivity), though. While the standard analyses suggest considerable heritability, most GCTA estimates are zero or close to zero. Here's the result for self-report measures, with standard results on the left and GCTA results on the right:

Results for parent and teacher reports are broadly similar.

What's it all mean? Well, the authors include a long discussion section in their paper, but, frankly, much of it is above my head due to my very limited knowledge of genetics and associated research methods. The most important take-home message, though, is that at least one of the common methods for estimating the contribution of genes to human outcomes yields misleading results. This is very important, and it is to be hoped that the paper gets lots of exposure. I've done my part.

Pebbles, Vol. 44

1. When absence of evidence is evidence of absence (xkcd)

2. An attempt to answer the question, "What were Stanley Kubrick's favourite films?" (Nick Wrigley)

3. One more time: Changing football coaches seems to make no sense (on average, within the observed range) (Hannah Barnes)

4. Where does the phrase "I, for one, welcome our new insect overlords" come from?

5. Researching the effect of stop-and-frisk is really hard. (Benjamin Wallace-Wells)

6. Andrew Gelman on Axelrod's interpretive overreach concerning trench warfare and the multiple comparisons problem and expressing uncertainty.

7. Who guards the guardians? Cameras could. (Randall Stross)

8. "Slot machine sounds can manipulate players, researchers say" (Meerie Kim)

Around the Blogs, Vol. 101: Long Wait, Long List

Because I've been collecting for so long, it's so many links. Because it's so many links, I'm posting it early.

1. If the effect in question was found in a particularly small sample, should that strengthen or weaken your belief in the effect? (Eric Falkenstein) From the same author: A critique of Stevenson and Wolfers' happiness research.

2. Thoughtful, personal essay by Eric S. Raymond about the emotion and cognition of racism.

3. A body-mind theory of lefties and righties (Agnostic)

4. "Annals of Self-Refuting Tweets" (Jeremy Freese presents the American Sociological Association make an ass of itself)

5. Wie intensiv werden die Deutschen eigentlich von der eigenen Regierung ausgespäht? Man weiß es nicht. (Niko Härting) (via)

6. "A conservative estimate is that we’re spending a million dollars per year per terrorist, maybe more – that’s not even counting Iraq and Afghanistan." (Gregory Cochran)

7. The case against (eating lunch) outside (Matthew Yglesias) (via)

8. Matthew Desseem reviews Rififi.

9. Person fixed effects and psychological testing.

10. The theory that Marcia Lucas contributed more to Star Wars' quality than is usually acknowledged. (Fabio Rojas)

11. A discussion of reviewing and reviewers (with a focus on sociology) (olderwoman and commenters)

12. Is US violent crime actually down? Looking at non-police data. (Steve Sailer)

13. "William Boyd’s Taxonomy of the Short Story" (Will Wilkinson)

14. How not to get published. (Andrew Gelman/Brian Nosek, Jeffrey Spies, and Matt Motyl)

15. Getting the priorities straight (Foseti) (on this blog)

16. Male feminists: Demand and supply. (Nick Borman)

17. Real life cases of amnesia that are stranger than fiction. (Christian Jarrett)

18. Season of birth is endogenous (Eric Crampton/Kasey S. Buckles and Daniel M. Hungerman)

19. A model of how the internet works (Marco Arment) (via)

Around the Blogs, Vol. 100: My Mama, She Don't Told Me, My Papa Don't Told Me, Too

1. Four types of story elements, and why one may be particularly popular in rich societies (Robin Hanson/Orson Scott Card)

2. Trying to infer psychological events from neural activations (Neuroskeptic) (via)

3. The limits of stereotype accuracy (Bryan Caplan/Clark McCauley)

4. The correct way to argue with Milton Friedman (Daniel Davies) (see also)

5. Now that's what I call selection effects! (Andrew Gelman/Sharon Begley and Robin Respaut)

Taubes on the Limits of Epidemiology

Here's a very interesting 2007 article by Gary Taubes, on the limits of epidemiology. One interesting bit:

The subjects were some 8,500 middle-aged men with established heart problems. Two-thirds of them were randomly assigned to take one of the five drugs and the other third a placebo. Because one of the drugs, clofibrate, lowered cholesterol levels, the researchers had high hopes that it would ward off heart disease. But when the results were tabulated after five years, clofibrate showed no beneficial effect. The researchers then considered the possibility that clofibrate appeared to fail only because the subjects failed to faithfully take their prescriptions.

As it turned out, those men who said they took more than 80 percent of the pills prescribed fared substantially better than those who didn’t. Only 15 percent of these faithful “adherers” died, compared with almost 25 percent of what the project researchers called “poor adherers.” This might have been taken as reason to believe that clofibrate actually did cut heart-disease deaths almost by half, but then the researchers looked at those men who faithfully took their placebos. And those men, too, seemed to benefit from adhering closely to their prescription: only 15 percent of them died compared with 28 percent who were less conscientious. “So faithfully taking the placebo cuts the death rate by a factor of two,” says David Freedman, a professor of statistics at the University of California, Berkeley. “How can this be? Well, people who take their placebo regularly are just different than the others. The rest is a little speculative. Maybe they take better care of themselves in general. But this compliance effect is quite a big effect.”

And another:

Indeed, if you ask the more skeptical epidemiologists in the field what diet and lifestyle factors have been convincingly established as causes of common chronic diseases based on observational studies without clinical trials, you’ll get a very short list: smoking as a cause of lung cancer and cardiovascular disease, sun exposure for skin cancer, sexual activity to spread the papilloma virus that causes cervical cancer and perhaps alcohol for a few different cancers as well.

Note that it says "without clinical trials", though. He's also got answers to the question, "So how should we respond the next time we’re asked to believe that an association implies a cause and effect" that seem reasonable. Recommended for anyone interested in health and/or research methods and causality (but keep in mind I'm not an expert on medicine).

Estimating the Effect of Helmet Laws on Cycling-related Injuries: You Can't Do It Like That

In some places there are laws that require people to wear helmets when cycling. One may wonder what effects these regulations have on injuries. That's a question a paper (open access) by Jessica Dennis, Tim Ramsay, Alexis F. Turgeon and Ryan Zarychanski is trying to answer. They use data from the Canadian provinces, some of which introduced helmet legislation for minors only, while in other provinces the laws apply to people of all ages, and yet others introduced no such legislation. Have a look at the basic data:

Red lines are for adults, blue lines for minors. The dotted lines indicate when the legislation was introduced. You'll note that the provinces differ in when they introduced the laws. There are no clear breaks in the trends when the laws are introduced. On the other hand:

The rate of hospital admissions for cycling related head injuries in Canada among young people decreased from 17.0 to 4.9 per 100 000 person years between 1994 and 2008 (fig 1⇓). In provinces that implemented helmet legislation, the rate decreased steeply between 1994 and 2003, the time over which legislation was implemented, from 15.9 to 7.3 per 100 000 person years, corresponding to a 54.0% (95% confidence interval 48.2% to 59.8%) reduction. In provinces and territories that did not implement helmet legislation, the rate of admissions for cycling related head injuries also decreased between 1994 and 2003, but to a lesser degree. The reduction in provinces without legislation was 33.2% (23.3% to 43.0%), corresponding to a decrease from 19.1 to 12.9 per 100 000 person years. Among adults, the rate of admissions for cycling related head injuries was low in all provinces and across all study years. Between 1994 and 2003, the rate of head injuries in adults in provinces with helmet legislation decreased by 26.2% (16.0% to 36.3%), a reduction from 3.0 to 2.2 per 100 000 person years, compared with a negligible increase in rates in provinces and territories with no legislation, from 2.7 to 2.8 per 100 000 person years.

That's the authors' preliminary, narrative analysis. They point out that other cycling-related injuries also decreased. The authors then make some data analysis decisions which I would describe as suboptimal. First, they run an interrupted time series regression for each province separately, adjusting for trends. Second, they do not differentiate between provinces in which the laws apply only to minors and those where they apply to all, on the basis that some other study found spillover effects of legislation aimed at young people on helmet use in adults. Third, they take as their dependent variable hospital admissions for cycling-related head injuries as a ratio of hospital admissions for all cycling-related injuries.

The authors estimate no significant effects and conclude that "the incremental contribution of provincial helmet legislation to reduce the number of hospital admissions for head injuries is uncertain to some extent, but seems to have been minimal."

But you cannot conclude that from their analysis. First, recall that the provinces introduced their laws in different years. Dennis et al. throw that variation away and hence cannot control for time effects. Just pool the data and run a regression controlling for both province and year fixed effects! I guess that's almost all you need for identification, but one might consider controlling for differences in weather, which surely must have some effect on cycling.

Second, why not differentiate between laws applicable to all cyclists and minors only, respectively? Just use two different dummies. If the minors-only laws have effects on adults, that's information you want to explicate.

Third, and most importantly, you really, really do not want to adjust for all cycling-related injuries. The authors state that they do this in order to adjust for changes in cycling. But this makes no sense, and doubly so. (i) You automatically adjust-out any differences that the laws might make by reducing cycling. I believe there are studies suggesting such an effect, but I have not seen them. It would certainly make sense: Forcing people to wear a helmet makes cycling less attractive to some. (ii) There is a large literature on the topic of the consumption of risk (Peltzman effect). The idea is that when safety measures are put into place, people are going to consume some of that risk by adjusting their behaviour. For example, cyclists might cycle faster. So some of the effect of the law should be on cycling-related injuries not to the head.

In other words, this is an ideal design to find no effects even if there are some. I'm not saying that's deliberate - maybe it is more appropriate to say that this reflects disciplinary differences. For a medical researcher, it's probably natural to ask how much a helmet helps once there is an accident, which is roughly what the adjust-for-all-injuries strategy does. But if you measure that, you're not measuring the full effect of the law, which is the authors' stated aim. The concept of consumption of risk is standard knowledge in economics, and also known in other social sciences. And any undergraduate who has taken in, say, Wooldridge's Introductory Econometrics, should be able to suggest the design I outlined above, especially given the yummy data structure. Maybe that's just not obvious if your training was in medicine.

In this case, and as a noneconomist, I'll say it's the (hypothetical) economists who get it right. Oh, and I don't think you should use significance tests with this data.

Instrumental Variables: The Pina Colada Explanation

OLS can tell us about correlations in data, but won't be able to say much about causality. IV regressions turn observational data into a pseudo-experiment.

For example, suppose that we had data on courses taken and earnings, as in the study you cite above. Suppose simple OLS told us that people who take math courses earn more. That is fine as far as it goes, but we really want to know if it is just that smarter people take math courses, or that the math course itself increases earnings. OLS isn't going to help, but a good IV will.

Suppose that due to a bad pineapple, the pina colada mix at a mathematics department picnic in one high school was poisoned. Half of the mathematics faculty was laid up in the hospital for a semester. Many planned math courses were not offered that year. Presumably, the only way the pina colada disaster affected the future earnings of students is through the courses they were able to take. The exogenous variation in mathematics courses available creates the equivalent of an intent-to-treat experiment. Even the smart students were less likely to take math courses during the pina colada year.

Now, as in any statistical procedure, garbage in, garbage out. If the IV is invalid or weak, then the result of the IV regression is totally meaningless.

That's David Jinkins's comment on Bryan Caplan's post on instrumental variables. I don't think much of the post itself, but some of the reader contributions are quite good.

An aspect I would like to see mentioned more often in this context is that the impression conveyed by many articles - that 2SLS is the same as OLS, but with added causality - is quite simply false. Rather, if everything goes right, you are measuring the effect of the variance in the endogenous regressor as influenced by the variation in the instrument. This can be an advantage in some cases, but typically, you want to know about the target variable "as is", and you don't get that.

Gender and Causality

As far as I can see, a lot of the astronomical number of citations Paul Holland's article "Statistics and Causal Inference" has accumulated is due to the fact that he coined the term "fundamental problem of causality", describing the fact that you can observe a unit only with or without treatment, but not both. Every time you mention that problem, you're pretty much obliged to cite Holland. I'm not so happy with Holland's coinage of the term, because I would have thought that an even more fundamental problem of causality - or rather, the estimation of causal influences -  than the one Holland referred to is that you cannot observe causality.

Like many others, I am also unhappy with the maxim "No causation without manipulation" that Holland's article spread. Writes Markus Gangl in his pretty good (but not untechnical) review of causal inference with observational data (pp. 38-39; gated link):

The perception that the counterfactual framework would primarily apply to the effects of policy interventions or other explicitly manipulated (or at least manipulable) treatments is perhaps the single most important impediment to its more widespread adoption in sociology. This perception is a major misunderstanding on the part of sociologists (cf. also Heckman 2005, Moffitt 2005, Sobel 1998). Whether nonmanipulable factors such as gender, race, or class affect life courses is a perfectly sensible counterfactual question to begin with [...]. With respect to gender, for example, the counterfactual “manipulation” in question is the determination of fetal gender at inception, which, moreover, is plausibly random (Rubin 1986), so that its causal effect is directly identified from the comparison of mean life-course outcomes among men and women from, for example, the same birth cohort or country. In this specific case, and ignoring SUTVA [...], the main impediment to causal inference is not so much a lack of controls, as a lack of representative samples (see Sobel 1998).

That's right: If you want to look at the causal effects of gender, just compare group means. I don't know, though, what Gangl is getting at with his remark on representative data - there's quite a few representative datasets that contain both men and women. The problem (that Gangl hints at in technical, general terms on p. 39) is rather that nobody really cares about the total effect of assignment to a sex. What people care about are the mediating mechanisms - testosterone, discrimination, that kind of thing.

Speaking of discrimination, feminism is a bewilderingly imprecise term, but I have found the following, very reductionist, model helpful when thinking about the strand sometimes called "radical feminism." You can see it as an ideology that took two figures of thought referring to the interrelationship between groups and applying them to gender. From anti-racism, radical feminism took the idea that differences between groups cannot be based on biological differences. From Marxism came the idea that history is mainly the struggle between groups, which leads to fanciful statements such as the description of rape as "nothing more or less than a conscious process of intimidation by which all men keep all women in a state of fear."

Note that the two combine nicely to shut off uncertainty. The no-biology view tells us that all observed differences between men and women must be due to discrimination of one sort or another; the men-against-women view tells us who's doing the discriminating. No further research needed.

Comments, Journal Space, the Internet, and the Cynical Theory of Journals

Andrew Gelman, discussing scientific standards in the social sciences, relates the following:

Recently I sent a letter to the editor to a major social science journal pointing out a problem in an article they’d published, they refused to publish my letter, not because of any argument that I was incorrect, but because they judged my letter to not be in the top 10% of submissions to the journal. I’m sure my letter was indeed not in the top 10% of submissions, but the journal’s attitude presents a serious problem, if the bar to publication of a correction is so high. That’s a disincentive for the journal to publish corrections, a disincentive for outsiders such as myself to write corrections, and a disincentive for researchers to be careful in the first place.

Commenter WB wonders:

Why don’t journals simply post serious criticisms and important corrections on their websites? Any reluctance to admit mistakes and publish corrections seems inexcusable given how easy it is to post items online. Obviously, websites don’t face the strict space limitations of print journals. So online sections could be used to publish items that aren’t “in the top 10% of submissions to the journal,” but are nonetheless important and worth the attention of readers.

And Gelman replies:

I suppose one reason they don’t do it is that it would take effort and expense to set up the website. Another difficulty is the need to review the critiques. If it were easier to publish a letter to the editor, I suppose the journal would get more submissions, then they’d need to find more reviewers, etc.

Yeah, maybe. But for the time being I'll hypothesize that the cynical theory of journals explains a larger chunk of the variances.

Around the Blogs, Vol. 93

1. Why social science research is so hard: The case of guns and violence. Part one, part two. (Maggie Koerth-Baker)

2. Important questions that are easy to not ask dept.: Which type of growth trajectory are you on? (Scott H. Young) And are there sudden jumps? (Ben Casnocha)

3. Brits probably shouldn't worry about the economic effects of immigration. (Tim Harford)

4. Self-reports underestimate BMIs in Ireland, too. (Economic Logician/David Madden)

5. Placebo effects in priming. (Christian Jarrett/Ulrich W. Weger & Stephen Loughnan) Will it replicate?

Which Economic Literatures Are the Least Trustworthy? Does Science Put Too Much Weight on the Base Rate?

Those tables (click to enlarge) are from an important paper called "Are All Economic Facts Greatly Exaggerated? Theory Competition and Selectivity" (gated, via) by Chris Doucouliagos and T.D. Stanley that's recently been published in the Journal of Economic Surveys (rather than the AER, where it probably belongs). The higher the beta-value, the more the publications in that literature are estimated to be biased due to selection (what gets published and what doesn't). Hence, the higher the value, the more the literature as a whole exaggerates how homogenous real-world phenomena are. Hypothetical example (mine, not theirs): Imagine you knew with certainty that, on average, a woman's colour of hair had no effect on how attractive men find her. Further imagine that the economic literature consistently showed that "gentlemen prefer blondes". You would then expect this literature to receive a high beta-value in the table. (The computation of the value is somewhat complicated, but based on the idea that literatures are selective if they feature many results that are just significant.)

The authors go on to estimate what predicts beta. Here's the paper's abstract:

There is growing concern and mounting evidence of selectivity in empirical economics. Most empirical economic literatures have a truncated distribution of results. The aim of this paper is to explore the link between publication selectivity and theory contests. This link is confirmed through the analysis of 87 distinct empirical economics literatures, involving more than three and a half thousand separate empirical studies, using objective measures of both selectivity and contests. Our meta–meta-analysis shows that publication selection is widespread, but not universal. It distorts scientific inference with potentially adverse effects on policy making, but competition and debate between rival theories reduces this selectivity and thereby improves economic inference.

Besides being a very important contribution on the trustworthiness of different literatures, this addresses a question that I've been thinking about quite a bit: Everybody knows that implausible results get double-checked more often than plausible ones (I once helped a friend who had found, using matching, that the results were exactly the opposite of what you should expect. Can you guess the reason?). So results that seem plausible get a leg up. Isn't that an unfair advantage for plausible results? Well, that depends on how good your prior theories are. If they're really good, then confirming results should get a leg up. But how do you know that they're good? Only by looking at empirical results. Etc., ad infinitum.

The authors run a regression to see what predicts the selectivity of literatures. It turns out that selectivity is higher when there is only one reigning theory - that is, when available theory makes a clear prediction on which way the results ought to go, the estimates are particularly untrustworthy. This suggests that results that conform to theory are given too much of a leg up, probably due to a number of processes including, but not limited to, double-checking. This, it seems to me, is a very, very important result.

The Best Blog Posts of 2012: Vanity Edition

After my post on other people's 2012 posts that I liked best, here are ten of my posts from last year that I find more worthwhile than the others. In chronological order:

• How "Social Distance" Is Misleading
• The within-between Fallacy
• The Discrimination Fallacy
• The Argument from the Absence of Apocalypse
• Poverty and Life Outcomes: A Theory of the Changing Relative Importance of Resources and Personality
• Why You Shouldn't Borrow Books from Acquaintances
• The Greatest Tracks of the 1990s, Part 3: 30-1
• Testing vs. Matching and the Attraction of the Aphorism
• Sometimes You Need Both Blades of the Scissors
• Why Most Studies of Male-Female Differences Are Useless

This is the last installment of my turn-of-the-year flurry in linklists. I promise the next post will contain substantial blocks of new text written by me.

The Best Blog Posts of 2012

Hmmmm . . . I had had the feeling that it had not been a particularly strong year for blog posts - so only ten this time, to keep up this series' ridiculously high standards.

As last time, brackets are appended to each link to indicate whether the post is Long, Medium lenght or Short; High-Brow, Mid-Brow or Low-Brow, and Funny or Not.

For the rest of this series, use the tag.

10. Scatterplot: "Eight Observations on 'Biology' and Social Science", by Jeremy Freese (L; HB; N)

9. iSteve: "Feminists: Still Making Children Cry on Christmas Morning", by Steve Sailer (L; MB; F)

8. Seth's Blog: "Two Dimensions of Economic Growth: GDP and Useful Knowledge", by Seth Roberts (M; HB; N)

7. EconLog: "My Beautiful Bubble", by Bryan Caplan (M; MB; N)

6. Meteuphoric: "Value Realism", by Katja Grace (L; HB; N)

5. Overcoming Bias: "Unspeakable Arrogance", by Robin Hanson (M; HB; F)

4. Cheap Talk: "Dogmatic Doesn’t Have To Mean Closed-Minded", either by Jeff Ely or by Sandeep Baliga, I don't know which (M; MB; N)

3. The Atlantic Business: "What Is Causality?", by Jim Manzi (L; HB; N)

2. Old School Panini: "The Lumberjack’s Top Ten", by Alex Bourof (M; LB; F)

1. Jaltcoh: "Why do people say, 'Life is too short?'", by John Althouse Cohen (S; MB; F)

Hats off to everyone on the list, and may everybody have a great year! "Everybody" includes people not on the list.

Around the Blogs, Vol. 89

Yeah, not a lot of original content lately; and given that it's the end of the year, readers should prepare for lots more listed-links-type of posts. Perhaps I'll get round to writing some original stuff over the holidays, but I doubt it. So, here's some noteworthy recent blog posts:

1. Steve Sailer vs. Elizabeth Sweet on economic development, sex roles, and presents. Funny and thought-provoking. From the same author: The Wertherpedia theory of mass shootings

2. Ben Casnocha's best Tweets collected

3. When outcomes for the treatment group affect outcomes for the control group, results won't scale up. (Economic Logician/Pieter Gautier, Paul Muller, Bas van der Klaauw, Michael Rosholm and Michael Svarer)

4. Perform your own meta-analysis (soon, perhaps) (passed along by Tyler Cowen)

5. A partial model of U.S. politics, from Andrew Hammel

Happy holidays, everyone.

Around the Blogs, Vol. 88: Social Science Special

1. Bryan Caplan discusses Malcolm X's autobiography: Directory to all parts.

2. How important are survey response rates? Jeremy Freese, Fabio Rojas.

3. Retracted psychology articles keep getting cited as evidence. (Sanjay Srivastava)

4. What would an ideal stats text book look like? (Phil Price)