Estimating the Effect of Helmet Laws on Cycling-related Injuries: You Can't Do It Like That

In some places there are laws that require people to wear helmets when cycling. One may wonder what effects these regulations have on injuries. That's a question a paper (open access) by Jessica Dennis, Tim Ramsay, Alexis F. Turgeon and Ryan Zarychanski is trying to answer. They use data from the Canadian provinces, some of which introduced helmet legislation for minors only, while in other provinces the laws apply to people of all ages, and yet others introduced no such legislation. Have a look at the basic data:

Red lines are for adults, blue lines for minors. The dotted lines indicate when the legislation was introduced. You'll note that the provinces differ in when they introduced the laws. There are no clear breaks in the trends when the laws are introduced. On the other hand:
The rate of hospital admissions for cycling related head injuries in Canada among young people decreased from 17.0 to 4.9 per 100 000 person years between 1994 and 2008 (fig 1⇓). In provinces that implemented helmet legislation, the rate decreased steeply between 1994 and 2003, the time over which legislation was implemented, from 15.9 to 7.3 per 100 000 person years, corresponding to a 54.0% (95% confidence interval 48.2% to 59.8%) reduction. In provinces and territories that did not implement helmet legislation, the rate of admissions for cycling related head injuries also decreased between 1994 and 2003, but to a lesser degree. The reduction in provinces without legislation was 33.2% (23.3% to 43.0%), corresponding to a decrease from 19.1 to 12.9 per 100 000 person years. Among adults, the rate of admissions for cycling related head injuries was low in all provinces and across all study years. Between 1994 and 2003, the rate of head injuries in adults in provinces with helmet legislation decreased by 26.2% (16.0% to 36.3%), a reduction from 3.0 to 2.2 per 100 000 person years, compared with a negligible increase in rates in provinces and territories with no legislation, from 2.7 to 2.8 per 100 000 person years.
That's the authors' preliminary, narrative analysis. They point out that other cycling-related injuries also decreased. The authors then make some data analysis decisions which I would describe as suboptimal. First, they run an interrupted time series regression for each province separately, adjusting for trends. Second, they do not differentiate between provinces in which the laws apply only to minors and those where they apply to all, on the basis that some other study found spillover effects of legislation aimed at young people on helmet use in adults. Third, they take as their dependent variable hospital admissions for cycling-related head injuries as a ratio of hospital admissions for all cycling-related injuries.

The authors estimate no significant effects and conclude that "the incremental contribution of provincial helmet legislation to reduce the number of hospital admissions for head injuries is uncertain to some extent, but seems to have been minimal."

But you cannot conclude that from their analysis. First, recall that the provinces introduced their laws in different years. Dennis et al. throw that variation away and hence cannot control for time effects. Just pool the data and run a regression controlling for both province and year fixed effects! I guess that's almost all you need for identification, but one might consider controlling for differences in weather, which surely must have some effect on cycling.

Second, why not differentiate between laws applicable to all cyclists and minors only, respectively? Just use two different dummies. If the minors-only laws have effects on adults, that's information you want to explicate.

Third, and most importantly, you really, really do not want to adjust for all cycling-related injuries. The authors state that they do this in order to adjust for changes in cycling. But this makes no sense, and doubly so. (i) You automatically adjust-out any differences that the laws might make by reducing cycling. I believe there are studies suggesting such an effect, but I have not seen them. It would certainly make sense: Forcing people to wear a helmet makes cycling less attractive to some. (ii) There is a large literature on the topic of the consumption of risk (Peltzman effect). The idea is that when safety measures are put into place, people are going to consume some of that risk by adjusting their behaviour. For example, cyclists might cycle faster. So some of the effect of the law should be on cycling-related injuries not to the head.

In other words, this is an ideal design to find no effects even if there are some. I'm not saying that's deliberate - maybe it is more appropriate to say that this reflects disciplinary differences. For a medical researcher, it's probably natural to ask how much a helmet helps once there is an accident, which is roughly what the adjust-for-all-injuries strategy does. But if you measure that, you're not measuring the full effect of the law, which is the authors' stated aim. The concept of consumption of risk is standard knowledge in economics, and also known in other social sciences. And any undergraduate who has taken in, say, Wooldridge's Introductory Econometrics, should be able to suggest the design I outlined above, especially given the yummy data structure. Maybe that's just not obvious if your training was in medicine.

In this case, and as a noneconomist, I'll say it's the (hypothetical) economists who get it right. Oh, and I don't think you should use significance tests with this data.

No comments: