The British Medical Journal published a new study that claims eating chocolate can reduce your risk for irregular heart rhythms. You can expect to see a bunch of hype on the Internet today about how chocolate is good for you! So, is it time to have chocolate bars for dinner? Hold on a minute.
The quality of science in the medical field is a bit of a pet peeve of mine. It’s not entirely the fault of the scientists; I mean, a real controlled experiment that involves life or death can be tricky to say the least. But there are real problems:
- Lying with statistics – experiments often suffer from small sample sizes, or are not controlled. If it’s not a double-blind study, or it’s not a study with a large number of participants, the results must be taken with a grain of salt. All too often, the media reports on studies published from journals that don’t have high standards for experimental design.
- Funding bias / conflict of interests – all too often, a sensational scientific claim turns out to be funded by an organization that stands to benefit from it. A recent study about how cheese isn’t bad for you made the rounds on the Internet, but it was funded by the Global Dairy Platform and the Dairy Research Institute.
- The reproducibility crisis or replication crisis – over half of published studies cannot be replicated, and in some fields it’s much higher.
- Publication bias – researchers publish because they want tenure, and have a bias to only publish successful experiments as a result. We never hear about negative results.
These are huge issues that threaten the credibility of science as a whole. So what’s a person to do when faced with a claim like “chocolate’s good for your heart?”
Well, it’s on us to apply critical thinking and dig into the details. Let’s ask a few questions about this study:
Are they lying with statistics? This paper claims a 10% risk reduction in people who eat chocolate. 10% isn’t that big of a number, so I’d expect to see a large sample size (how many people were studied) before taking that seriously. Turns out this study had 55,000 participants, so a 10% difference does seem significant. You don’t need a degree in statistics to understand this – even if you rigorously measure statistical confidence, in the end it’s still a judgment call – statistics can only say how likely a measured effect is to be real; it never says it’s real conclusively.
Is there a conflict of interests? In this case, no. This study wasn’t funded by anyone who stands to benefit from increased sales of chocolate. This test also checks out on this study.
Has it been reproduced? No, but that’s not unusual for new research. However, if you really plan on increasing your chocolate consumption as a result of this study, it would be prudent to wait a few months and see if anyone has attempted to reproduce these findings (or proven unable to do so.) If I were a betting man, I’d say it won’t happen. Thanks to publication bias, it may take quite a bit of digging to find these results on your own.
Was it a controlled experiment? No, and the authors of the paper freely admit this. It was not a “double-blind” study; it merely looked for correlations between people who ate chocolate and those who didn’t with irregular heart rhythms. Correlation does not imply causality – there may have been some other underlying difference between these two groups that occurred randomly. Or perhaps there is some connection between eating chocolate and the real cause of the difference – perhaps chocolate-eaters are more likely to be physically active, in an effort to burn off that extra fat. This is the main issue with this particular study, and why you should take it with a huge grain of salt. Salted chocolate, as it were.
In the end, I wouldn’t call this particular study “bad science,” but its results should be taken with a good helping of skepticism – and the study’s authors admit as much, which good scientists should do. It’s not a controlled experiment, the results aren’t overly dramatic, and it hasn’t been reproduced yet.
Still, a chocolate bar sounds pretty good right now.