Bad Science Reporting

(Sorry, I know I promised to describe an experiment that would give evidence for different theories of consciousness, but I want to do a quick rant about something else first. The consciousness/anthropics post is coming soon.)

From The Economist: Spare the rod: Spanking makes your children stupid

The article cites “nearly 30 studies from various countries” that “show that children who are regularly spanked become more aggressive themselves” and are “more likely to be depressed or take drugs.” And most relevant to the title of the article, another large study found that “young children in homes with little or no spanking showed swifter cognitive development than their peers.”

Now ask yourself, is it likely that the studies they are citing actually provide evidence for the causal claim they are using to motivate their parenting advice?

To do so, you would need a study involving some type of intervention in parenting behavior (either natural or experimental). That is, you would want a group of researchers who randomly select a group of parents, and tell half of them to beat their kids and the other half not to. Now, do you suspect that these are the types of studies the Economist is citing?? I think not. (I hope not…)

Maybe they got lucky and found a historical circumstance in which there was a natural intervention (one not induced by the experimenters but by some natural phenomenon), and found data about outcomes before and after this intervention. But for this, we’d have to find an occasion where suddenly a random group of parents were forced to stop or start beating their children, while another random group kept at it without changing their behavior. Maybe we could find something like this (like if there were two nearby towns with very similar parenting habits, and one of them suddenly enacted legislation banning corporal punishment), but it seems pretty unlikely. And indeed, if you look at the studies themselves, you find that they are just standard correlational studies.

Maybe they were able to control for all the major confounding variables, and thus get at the real causal relationship? But… really? All the major confounding variables? There are a few ways to do this (like twin studies), but the studies cited don’t do this.

Now, maybe you’re thinking I’m nitpicking. Sure, the studies only find correlations between corporal punishment and outcomes, but isn’t the most reasonable explanation of this that corporal punishment is causing those outcomes?

Judy Rich Harris would disagree. Her famous book The Nurture Assumption looked at the best research attempting to study the causal effects of parenting style on late-life outcomes and found astoundingly little evidence for any at all. That is, when you really are able to measure how much parenting style influences kids’ outcomes down the line, it’s hard to make out any effect in most areas.  And anyway, regardless of the exact strength of the effect of parenting style on later-life outcomes, one thing that’s clear to me is that it is not as strong as we might intuitively suspect. We humans are very good at observing correlation and assuming causation, and can often be surprised at what we find when rigorous causal studies are done.

Plus, it’s not too hard to think of alternative explanations for the observed correlations. To name one, we know that intelligence is heritable, that intelligence is highly correlated with positive life outcomes, and that poor people practice corporal punishment more than rich people. Given these three facts, we’d actually be more surprised if we found no correlation between intelligence and corporal punishment, even if we had no belief that the latter causes the former. And another: aggression is heritable and correlated with general antisocial behavior, which is in turn correlated with negative life outcomes. And I’m sure you can come up with more.

This is not really the hill I want to die on. I agree that corporal punishment is a bad strategy for parenting. But this is not because of a strong belief that in the end spanking leads to depression, drug addiction, and stupidity. I actually suspect that in the long run, spanking is pretty nearly net neutral; the effects probably wash out like most everything else in a person’s childhood. (My prior in this is not that strong and could easily be swayed by seeing actual causal studies that report the opposite.) There’s a much simpler reason to not hit your kids: that it hurts them! Hurting children is bad, so hitting your kids is bad; QED.

Regardless, what does matter to me is good science journalism, especially when it involves giving behavioral advice on the basis of a misleading interpretation of the science. I used to rail against the slogan “correlation does not imply causation”, as, in fact, in some cases correlational data can prove causal claims. But I now have a better sense of why this slogan is so important to promulgate. The cases where correlation proves causation are a tiny subset of the cases where correlation is claimed to prove causation by overenthusiastic science reporters unconcerned with the dangers of misleading their audience. I can’t tell you how often I see pop-science articles making exactly this mistake to very dramatic effect (putting more books in your home will raise your child’s IQ! Climate change is making suicide rates rise!! Eating yogurt causes cancer!!!)

So this is a PSA. Watch out for science reporting that purports to demonstrate causation. Ask yourself how researchers could have established these causal claims, and whether or not it seems plausible that they did so. And read the papers themselves. You might have to work through some irritating academese, but the scientists themselves typically do a good job making disclaimers like NOTICE THAT WE HAVEN’T ACTUALLY DEMONSTRATED A CAUSAL LINK HERE. (These are then often conveniently missed by the journalists reporting on them.)

For example, in the introduction to the paper cited by the Economist the authors write that “we should be very careful about drawing any causal conclusions here, even when there are robust associations. It is very likely that there will be other factors associated with both spanking and child outcomes. If certain omitted variables are correlated with both, we may confound the two effects, that is, inappropriately attribute an effect to spanking. For example, parents who spank their children may be weaker parents overall, and spanking is simply one way in which this difference in parenting quality manifests itself.”

This is a very explicit disclaimer to miss and then to go on writing a headline that gives explicit parenting advice that relies on a causal interpretation of the data!