How science works

My previous post discussed the myths surrounding the “replication crisis” in psychology/neuroscience research. As usual, it became way too long and I didn’t even cover several additional points I wanted to mention. I will leave most of these for a later post in which I will speculate about why failed replications, papers about incorrect/questionable procedures, and other actions by the Holy Warriors for Research Truth cause such a lot of bad blood. I will try to be quick in that one or split it up into parts. Before I can get around to this though, let me briefly (and I am really trying this time!) have a short intermission with practical examples of the largely theoretical and philosophical arguments I made in previous posts.

Science is self-correcting

I’ve said it before but it deserves saying again. Science self-corrects, no matter how much the Crusaders want to whine and claim that this is a myth. Self-correction isn’t always very fast. It can take decades or even centuries. When thoroughly incorrect theories take root, it can take some extraordinary effort to uproot them again. However, given the pursuit of truth and free expression continues unhindered and as long as research has sufficient resources and opportunities, a correction will eventually happen. In many ways self-correction is like evolution, climate change, or plate tectonics. You rarely see direct evidence for these slow changes by looking at single time points but the trends are clearly visible. Fortunately though, scientific self-correction is typically faster than these things.

Take the discovery last year of gravitational waves supporting the Big Bang theory of the origin of the universe. It was widely reported in the news media as a earth- (well, space-) shattering finding that would greatly advance our understanding of our place in the cosmos. It resulted in news articles, media reports, and endless social media shares of videos and articles about this and the Daily Mail immediately making a complete fool of itself as usual by revealing just how little they know about the scientific process. It sparked some discussion about sexism in science journalism because initial reports ignored one of the female scientists who inspired the theory. (I am too lazy to provide any links to these things. If you’re on this blog I assume you’ve mastered the art of google).

However, in the months that followed critical voices began to be heard suggesting that this discovery may have been a fluke. Instead of gravitational waves revealing the fabric of the universe itself, it appears that these measurements were contaminated by signals stemming from cosmic dust. Just the other day, Nature published a story suggesting that this alternative, simpler explanation now looks to be correct. I am not the Devil’s Astrophysicist so I can’t talk about the specifics of this research but it certainly sounds compelling from my laywoman’s  perspective: even without in-depth understanding of the research I can tell that the evidence suggests a simpler explanation for the initial findings and by Occam’s Razor alone I would be encouraged to accept this one as more probable.

They apparently also have orientation columns in space

I don’t see any problem with this and I am pretty sure neither do most scientists. This is just self-correction in action. If anything about this story has been problematic, it is the fact that the initial discovery was so widely publicized before it was validated. I don’t really know where I stand on this. On the one hand, this has been unfortunate because it could greatly confuse public opinion about this research. The same applies to the now “debunked” finding of arsenic life forms reported a few years ago. A spectacular, possibly paradigm-shifting discovery was reported widely only to be disproved at a later point. I could see that this sort of perceived instability in scientific discoveries could undermine the public’s trust in science with potentially devastating consequences. In a world where climate change deniers, “anti-vaxers”, and creationists are given an undeservedly strong platform to spread their views (no linking to those either for that very reason), we need to be very careful how science is communicated. A lack of faith in scientific research could do a lot of harm.

However, in my mind this doesn’t mean we should suppress media reports of new, possibly not-yet-validated discoveries. For one thing, in a free society we can’t forbid people from talking about their research. I also believe that science should be a leading example of democratic ideals and transparency because good science is based on it. Instead of strong regulation and rigid structures science flourishes when authority is questioned.

“[Science] connects us with our origins, and it too has its rituals and its commandments. Its only sacred truth is that there are no sacred truths. All assumptions must be critically examined. Arguments from authority are worthless.”

Carl Sagan – Cosmos, Episode 13 “Who speaks for Earth?”

More importantly, I believe as scientists we actually want to communicate the enthusiasm and joy we experience when we reveal secrets of the world – even if they are untrue. Rather than curtailing when and how we communicate our discoveries to the public, we should do a better job at communicating how science works in practice. By all means, we should improve our press releases to minimize the sweeping generalizations and unfounded speculations by the news media about the implications of a new finding. When communicating sensational results to the public, we should certainly convey our excitement. However, we should also always remind them (and ourselves) that any new discovery is never the last word on an issue but the first.

In my view helping the public understand that scientific knowledge is ever evolving and that “boffins” didn’t just “prove” things will in fact strengthen the public’s trust in science. The reason I, as a neuroscientist without any deeper understanding of atmospheric physics, believe that man-made climate change is a problem is that an overwhelming majority of climate scientists believe the evidence supports this hypothesis. I know what scientists are like. They disagree all the damn time about the smallest issues. If there is an overwhelming consensus on any point it is past time we listen!

The Psychoplication Crisis

After having talked about gravitational waves and arsenic bacteria, you can bet some smartass will come out of the woodwork and tell you that these are the “hard” natural sciences but the same does not apply to “soft” science like psychology or cognitive neuroscience. Within psychology/cognitive neuroscience the same snobbery exists between experimental psychology and social cognition researchers. Perhaps that is somewhat understandable as some of the first perception researchers were in fact physicists (this is the origin of the term psychophysics). However, I think this sort of segregation is delusional and the self-satisfied belief that your own research area is somehow a “harder” science can be quite dangerous because it makes it all that much easier to mislead you into thinking that your own research is not susceptible to these problems.

I don’t think the problems that are currently discussed in psychology and neuroscience are in any way specific to or even particularly pervasive in our field. Some concepts and theories are simply more established in physics than they are in psychology. We certainly don’t need to do any experiments to prove gravity or even to confirm many of the laws that seem to govern it. The same cannot be said about social psychology where people still question whether things like unconscious priming of complex behaviors exist in the first place. But the fact that sensational findings are hyped up by media reports only to be disproved in spectacular failures to replicate is evidently just as common in physics and microbiology. And even shocking revelations of scientific fraud have happened in those fields (see also here for a less clear-cut case – I’ll probably discuss this more next time).

One way in which the culture in physics research seems to differ from our field is that it is far more common to upload manuscripts to public repositories prior to peer review. This allows a wider, more public discussion of the purported discoveries which is certainly good for the scientific process. However, it also results in broader media coverage before the research has even passed the most basic peer review stage and this can in fact exacerbate the problems with unverified findings. The still rare cases of this approach in psychology research suffer from the same problem. Just look at Daryl Bem’s meta-analysis of “precognition” experiments which is available on such a repository even though it has not been formally published by a peer reviewed journal. Unlike his earlier findings, which were also widely discussed before they were published officially, this analysis hasn’t been picked up by many news outlets. However, it easily could have been, just as was the case for the earlier examples from physics and microbiology research. Another famous parapsychologist also posted it on his blog of as evidence that “the critics need to rethink their position” – a somewhat premature conclusion if you ask me.

In theory, I don’t see a problem with having new, un-reviewed manuscripts publicly available provided that the surrounding peer discussion is also visible. The problem is that this is not always the case. Certainly the news coverage of the peer discussion usually doesn’t match the media circus surrounding the original finding. Taking Bem’s meta-analysis as an example, it obviously has been under closed peer review for at least the better part of a year now. I would love to actually see this peer review discussion as it is developing but – with one exception (to my knowledge) – so far this review has been for the eyes of the peer reviewers only.

9 thoughts on “How science works

  1. Dear Pen-du,
    I have recently begun to read your posts, and enjoy them. In particular I appreciate their gnawing yet playful style. For example, the set of concepts that you have identified as the “Holy Warriors for Research Truth”, I have represented with Pirsig’s “Church of Reason”. This similarity, I believe, hacks at the heart of something we may have in common.

    And so, in response to this post arguing for open peer review, I would invite you to critique my work at Polyhistornaut.

    “science flourishes when authority is questioned”
    – I absolutely agree with this, and would add that there is no need to question authority when the authority is correct.

    “A lack of faith in scientific research could do a lot of harm.”
    – Should one hold faith in scientific research?

    Kind regards,
    Bojan Keevill


    • Thanks for your comments. I will check out your blog. Not entirely sure I understand this comment:

      “…there is no need to question authority when the authority is correct”

      That sounds kind of Orwellian or like the justification for totalitarian leadership. I assume you didn’t intend this but anyways, we never know whether an authority is correct. This is the beauty of science. It is about questioning and challenging ideas: “Take nobody’s word for it.”

      But of course what we must also do is value the evidence. Just randomly challenging authority isn’t science. Challenge all you want but don’t expect anyone to buy it unless you have a good case.

      “Should one hold faith in scientific research?”

      My answer to this is yes. However, ‘Faith’ is a very complex word. Let me rephrase this as ‘confidence’. We can be confident that the scientific method works. It is the best method we have for revealing objective truths about the world.

      My point is that while we can trust the process we should not believe every single result because most results are false or at the very least incomplete.

      The Devil’s Neuroscientist
      (Note that SamPenDu is the name of my host’s lab. We are independent entities.)

      Liked by 1 person

  2. Self correcting? Maybe. Maybe not. But if so then on what time scale? Through post publication peer review that time scale can be greatly reduced. The lay press will just have to learn to adapt to the new process of review and refrain from the usual science sensationalism.


    • There is no ‘maybe’ about it. As I have argued in several of my posts, it is inherent to the scientific process. Science may get stuck in local minima and it may look like a random walk before converging on the truth – but given sufficient time and resources science will self-correct. As for the time scales, I have also said before that they can be glacial or geological. Yet scientists should be able to comprehend such slow processes more than anyone else.

      I totally agree though that better post-publication review could help to reduce these time scales. The distinction between post-publication review and crusading for true science isn’t always that clear but in general I believe science can only become better if it involves open and informed discourse.


      • “There is no ‘maybe’ about it. As I have argued in several of my posts, it is inherent to the scientific process. Science may get stuck in local minima and it may look like a random walk before converging on the truth – but given sufficient time and resources science will self-correct.”

        I have a huge problem with these statements. Who says science will self-correct?!?! Maybe certain false-positive findings will be left alone and no-one will investigate them any further. At that point you have an incorrect scientific record.

        Also, saying “given sufficient time and resources science will self-correct” is a statement that is very easy to use to wipe all problems with current day science under the rug: nothing to see here, move along, move along…We scientists know what’s best, don’t you worry your pretty little head about it…


Comments are closed.