My previous post discussed the myths surrounding the “replication crisis” in psychology/neuroscience research. As usual, it became way too long and I didn’t even cover several additional points I wanted to mention. I will leave most of these for a later post in which I will speculate about why failed replications, papers about incorrect/questionable procedures, and other actions by the Holy Warriors for Research Truth cause such a lot of bad blood. I will try to be quick in that one or split it up into parts. Before I can get around to this though, let me briefly (and I am really trying this time!) have a short intermission with practical examples of the largely theoretical and philosophical arguments I made in previous posts.
Science is self-correcting
I’ve said it before but it deserves saying again. Science self-corrects, no matter how much the Crusaders want to whine and claim that this is a myth. Self-correction isn’t always very fast. It can take decades or even centuries. When thoroughly incorrect theories take root, it can take some extraordinary effort to uproot them again. However, given the pursuit of truth and free expression continues unhindered and as long as research has sufficient resources and opportunities, a correction will eventually happen. In many ways self-correction is like evolution, climate change, or plate tectonics. You rarely see direct evidence for these slow changes by looking at single time points but the trends are clearly visible. Fortunately though, scientific self-correction is typically faster than these things.
Take the discovery last year of gravitational waves supporting the Big Bang theory of the origin of the universe. It was widely reported in the news media as a earth- (well, space-) shattering finding that would greatly advance our understanding of our place in the cosmos. It resulted in news articles, media reports, and endless social media shares of videos and articles about this and the Daily Mail immediately making a complete fool of itself as usual by revealing just how little they know about the scientific process. It sparked some discussion about sexism in science journalism because initial reports ignored one of the female scientists who inspired the theory. (I am too lazy to provide any links to these things. If you’re on this blog I assume you’ve mastered the art of google).
However, in the months that followed critical voices began to be heard suggesting that this discovery may have been a fluke. Instead of gravitational waves revealing the fabric of the universe itself, it appears that these measurements were contaminated by signals stemming from cosmic dust. Just the other day, Nature published a story suggesting that this alternative, simpler explanation now looks to be correct. I am not the Devil’s Astrophysicist so I can’t talk about the specifics of this research but it certainly sounds compelling from my laywoman’s perspective: even without in-depth understanding of the research I can tell that the evidence suggests a simpler explanation for the initial findings and by Occam’s Razor alone I would be encouraged to accept this one as more probable.
I don’t see any problem with this and I am pretty sure neither do most scientists. This is just self-correction in action. If anything about this story has been problematic, it is the fact that the initial discovery was so widely publicized before it was validated. I don’t really know where I stand on this. On the one hand, this has been unfortunate because it could greatly confuse public opinion about this research. The same applies to the now “debunked” finding of arsenic life forms reported a few years ago. A spectacular, possibly paradigm-shifting discovery was reported widely only to be disproved at a later point. I could see that this sort of perceived instability in scientific discoveries could undermine the public’s trust in science with potentially devastating consequences. In a world where climate change deniers, “anti-vaxers”, and creationists are given an undeservedly strong platform to spread their views (no linking to those either for that very reason), we need to be very careful how science is communicated. A lack of faith in scientific research could do a lot of harm.
However, in my mind this doesn’t mean we should suppress media reports of new, possibly not-yet-validated discoveries. For one thing, in a free society we can’t forbid people from talking about their research. I also believe that science should be a leading example of democratic ideals and transparency because good science is based on it. Instead of strong regulation and rigid structures science flourishes when authority is questioned.
“[Science] connects us with our origins, and it too has its rituals and its commandments. Its only sacred truth is that there are no sacred truths. All assumptions must be critically examined. Arguments from authority are worthless.”
Carl Sagan – Cosmos, Episode 13 “Who speaks for Earth?”
More importantly, I believe as scientists we actually want to communicate the enthusiasm and joy we experience when we reveal secrets of the world – even if they are untrue. Rather than curtailing when and how we communicate our discoveries to the public, we should do a better job at communicating how science works in practice. By all means, we should improve our press releases to minimize the sweeping generalizations and unfounded speculations by the news media about the implications of a new finding. When communicating sensational results to the public, we should certainly convey our excitement. However, we should also always remind them (and ourselves) that any new discovery is never the last word on an issue but the first.
In my view helping the public understand that scientific knowledge is ever evolving and that “boffins” didn’t just “prove” things will in fact strengthen the public’s trust in science. The reason I, as a neuroscientist without any deeper understanding of atmospheric physics, believe that man-made climate change is a problem is that an overwhelming majority of climate scientists believe the evidence supports this hypothesis. I know what scientists are like. They disagree all the damn time about the smallest issues. If there is an overwhelming consensus on any point it is past time we listen!
The Psychoplication Crisis
After having talked about gravitational waves and arsenic bacteria, you can bet some smartass will come out of the woodwork and tell you that these are the “hard” natural sciences but the same does not apply to “soft” science like psychology or cognitive neuroscience. Within psychology/cognitive neuroscience the same snobbery exists between experimental psychology and social cognition researchers. Perhaps that is somewhat understandable as some of the first perception researchers were in fact physicists (this is the origin of the term psychophysics). However, I think this sort of segregation is delusional and the self-satisfied belief that your own research area is somehow a “harder” science can be quite dangerous because it makes it all that much easier to mislead you into thinking that your own research is not susceptible to these problems.
I don’t think the problems that are currently discussed in psychology and neuroscience are in any way specific to or even particularly pervasive in our field. Some concepts and theories are simply more established in physics than they are in psychology. We certainly don’t need to do any experiments to prove gravity or even to confirm many of the laws that seem to govern it. The same cannot be said about social psychology where people still question whether things like unconscious priming of complex behaviors exist in the first place. But the fact that sensational findings are hyped up by media reports only to be disproved in spectacular failures to replicate is evidently just as common in physics and microbiology. And even shocking revelations of scientific fraud have happened in those fields (see also here for a less clear-cut case – I’ll probably discuss this more next time).
One way in which the culture in physics research seems to differ from our field is that it is far more common to upload manuscripts to public repositories prior to peer review. This allows a wider, more public discussion of the purported discoveries which is certainly good for the scientific process. However, it also results in broader media coverage before the research has even passed the most basic peer review stage and this can in fact exacerbate the problems with unverified findings. The still rare cases of this approach in psychology research suffer from the same problem. Just look at Daryl Bem’s meta-analysis of “precognition” experiments which is available on such a repository even though it has not been formally published by a peer reviewed journal. Unlike his earlier findings, which were also widely discussed before they were published officially, this analysis hasn’t been picked up by many news outlets. However, it easily could have been, just as was the case for the earlier examples from physics and microbiology research. Another famous parapsychologist also posted it on his blog of as evidence that “the critics need to rethink their position” – a somewhat premature conclusion if you ask me.
In theory, I don’t see a problem with having new, un-reviewed manuscripts publicly available provided that the surrounding peer discussion is also visible. The problem is that this is not always the case. Certainly the news coverage of the peer discussion usually doesn’t match the media circus surrounding the original finding. Taking Bem’s meta-analysis as an example, it obviously has been under closed peer review for at least the better part of a year now. I would love to actually see this peer review discussion as it is developing but – with one exception (to my knowledge) – so far this review has been for the eyes of the peer reviewers only.