Next Tuesday, St Patrick’s Day 2015, UCL Experimental Psychology will organize an event called “Is Science broken?”. It will start with a talk by Chris Chambers of Cardiff University about registered reports. Chris has been a vocal proponent of preregistration, making him one of the generals of the Crusade for True Science, and registered reports are the strictest of the preregistration proposals to date: it involves peer-review of the introduction and methods section of a study before data collection commences.
Chris’ talk will be followed by an open discussion by a panel comprising a frightening list of highly esteemed cognitive neuroscientists: in addition to Chris, there will be David Shanks (who’ll act as chair), Sophie Scott, Dorothy Bishop, Neuroskeptic … and – for some puzzling reason – my alter ego Sam Schwarzkopf. I am sure that much of the debate will focus on preregistration although I hope that there will also be a wider discussion of the actual question posed by the event:
“Is Science broken?” – Attentive readers of my blog will probably guess that my answer to this question is a clear No. In fact, I would question whether the question even makes any sense. Science cannot be broken. Science is just a method – the best method there is – to understand the Cosmos. It is inherently self-correcting, no matter how much Crusaders like to ramble against this notion. Science is self-correction. To say that science is broken is essentially stating that science cannot converge on the truth. If that were true, we should all just pack up and go home.
Now, we’ve been over this and I will probably write about this again in the future. I’ll spare you another treatise on self-correction and model fitting analogies. What the organizers of the event mean in reality is that the human endeavor of scientific research is somehow broken. But is that even true?
Sam is the one who’ll be attending that panel discussion, not me. I don’t know if he’ll let me out of my cage to say anything. We do not always see eye-to-eye as he has the annoying habit to always try to see things from other people’s perspectives… I think though that fundamentally he agrees with me on the nature of the scientific method and I hope he will manage to bring this point across. Either way, here are some questions I think he should raise:
I. The Crusaders claim that a majority of published research findings are false positives. So far nobody has given a satisfying answer to the question how many false positives should we tolerate? 5%? 1%? Or zero? Is that realistic?
II. The Crusader types often complain about high-impact journals like Nature and Science because they value “novelty” and high risk findings. This supposedly hurts the scientific ideal of seeking the truth. But is that true? Don’t we need surprising, novel, paradigm-shifting findings to move science forward? Over the years of watching Sam do science I have developed a strong degree of cynicism. The scientific truth is never as simple and straightforward as the narrative presented in research papers – including those in most low-impact journals. Science publishing should be about communicating results and hypotheses even if they are wrong – because in the long run most inevitably will be.
III. We can expect there to be at least some discussion of preregistration at the debate. This will be interesting because Sophie has been one of the few outspoken critics of this idea. I have of course also written about this before and it’s a shame that I can’t participate directly in this debate. We would make one awesome dynamic duo of science ladies raging against preregistration! However much Sam will let me say about this though, putting all the naive debate about the potential benefit and problems of this practice aside, the thing that has bothered me most about it is that we have no compelling evidence for or against preregistration. What is more, there also seems to be little consensus on how can we even establish whether preregistration works. To me these are fundamental concerns that we need to address now before this idea really takes off. Isn’t it ironic that given the topic of this debate, nobody seems to seriously talk about these points? Shouldn’t scientists value evidence above all else? Shouldn’t proponents of preregistration be expected to preregister the design of their revolutionary experiment?
IV. In addition to these rather lofty concerns, I also have a pragmatic question. Is the Crusaders’ behavior responsible? Even assuming that people do not engage in outright questionable research practices (and I remain unconvinced that they are as widespread as the Crusaders say), I think you can almost guarantee that most preregistered research findings will be lackluster compared to those published in the traditional model. In a world where the traditional model dominates, won’t that harm the junior researchers who are coerced to follow the rigid model? A quote I heard on the grapevine was “My supervisor wants me to preregister all my experiments and it’s destroying my career”. How do we address this problem?
As I said, Sam might not let me say these things and he doesn’t agree with me on many accounts. However, I hope there will be some debate around these questions. I will try to report back soon although Sam is demanding a lot of time for himself these days… Stay tuned.