The Older Paper: The Limits of Fixed Effect and Inverse Probability of Treatment Weighting (2009)

David Bjerk's "How Much Can We Trust Causal Interpretations of Fixed-Effects Estimators in the Context of Criminality?" is the kind of stats paper I like: applied, short on Greek letters, and written in a crystal-clear style. In it, Bjerk demonstrates the limits of fixed effects and inverse probability of treatment (aka inverse propensity score) weighting in terms of establishing causality. (In Bjerk's usage, IPT weights are a kind of fixed effect, which is kinda strange, as the IPT weights he uses are time-varying. But never mind.)

As for fixed effects proper, he simply illustrates something every reader of a textbook should have learned, but which is quickly forgotten in practice: fixed effects do not account for influences on the dependent variable which vary over time. His demonstration is rather neat, though. The background is that some papers use fixed effects estimation to establish the effect of an independent variable such as alcohol consumption on a dependent variable such as aggressive behaviour. He first replicates this effect in a sample of adolescents and then shows that, using this method, we can also "show" that smoking causes aggression. This seems implausible, however - if anything, nicotine consumption may be expected to reduce aggression. The plausible explanation for both results is that both the outcome and the supposed "treatments" change in response to unmeasured variables, such as adolescents' decisions to rebel.

The portion about IPT weighting is even more interesting. The author focuses on a well-known paper by Robert Sampson, John Laub and Christopher Wimmer who use dynamic IPT weighting to establish that marriage causes a reduction in criminal involvement. Again replicating this result, Bjerk goes on to argue that such results are not very trustworthy (my emphasis):

[O]ne must assume that whatever the changes are that induce an individual to obtain the treatment (i.e. marriage) at a given point in time have no direct effect on his underlying propensity to commit crime at that point in time. In other words, one must assume there is no dynamic selection. As discussed above, this assumption may be problematic with respect to marriage and crime [...].

The remaining question then is whether IPT weights lessen or exacerbate any dynamic selection bias? To answer this question it is again crucial to think very closely about the processes through which individuals obtain the treatment of interest and whether these processes may differ between those who have a high likelihood of obtaining the treatment at any given point in time based on their other observable characteristics, and those who do not. In particular, if one believes that the changes that induce individuals to obtain the treatment at a given point in time are more generally orthogonal to any concurrent changes in criminal propensity for those with a low likelihood of obtaining the treatment at any point in time than for those with a high likelihood, then IPT weighting will indeed lessen the dynamic selection bias of fixed-effects estimates. However, if one believes the opposite is true, then IPT weighting will exacerbate the dynamic selection bias of fixed-effects estimates.

[...] Consider marriage. What would prompt an individual whose observables suggest a high likelihood of getting married at a point in time to actually get married? The high empirical likelihood of marriage at a given point in time means that such an individual already has many of the characteristics of responsibility and adulthood (e.g. steady employment, a weak criminal history, perhaps already in a cohabiting relationship). Therefore, it actually seems reasonable to think that a very small event or change in attitudes could trigger a marriage decision for these men, and moreover such changes could very plausibly have nothing to do with changes in their underlying criminal propensity. On the other hand, for individuals with a low empirical likelihood of marriage at any given point in time, it is likely that reasonably large events or changes in attitudes must occur for them to enter into a marriage, and it is less clear that such large events or changes in attitudes would not also affect these individuals’ underlying criminal propensities. Hence, it is actually quite plausible that the sequential ignorability assumption is more problematic for the low likelihood of marriage men than the high likelihood of marriage men, meaning IPT weighting will increase the dynamic selection bias with respect to fixed-effects estimates of the direct effect of marriage on individual criminality.

That hadn't occured to me. Recommended, including for teaching (as a supplement to a textbook treatment of the techniques in question).