Is Science broken?

Next Tuesday, St Patrick’s Day 2015, UCL Experimental Psychology will organize an event called “Is Science broken?”. It will start with a talk by Chris Chambers of Cardiff University about registered reports. Chris has been a vocal proponent of preregistration, making him one of the generals of the Crusade for True Science, and registered reports are the strictest of the preregistration proposals to date: it involves peer-review of the introduction and methods section of a study before data collection commences.

Chris’ talk will be followed by an open discussion by a panel comprising a frightening list of highly esteemed cognitive neuroscientists: in addition to Chris, there will be David Shanks (who’ll act as chair), Sophie Scott, Dorothy Bishop, Neuroskeptic … and – for some puzzling reason – my alter ego Sam Schwarzkopf. I am sure that much of the debate will focus on preregistration although I hope that there will also be a wider discussion of the actual question posed by the event:

“Is Science broken?” – Attentive readers of my blog will probably guess that my answer to this question is a clear No. In fact, I would question whether the question even makes any sense. Science cannot be broken. Science is just a method – the best method there is – to understand the Cosmos. It is inherently self-correcting, no matter how much Crusaders like to ramble against this notion. Science is self-correction. To say that science is broken is essentially stating that science cannot converge on the truth. If that were true, we should all just pack up and go home.

Now, we’ve been over this and I will probably write about this again in the future. I’ll spare you another treatise on self-correction and model fitting analogies. What the organizers of the event mean in reality is that the human endeavor of scientific research is somehow broken. But is that even true?

Sam is the one who’ll be attending that panel discussion, not me. I don’t know if he’ll let me out of my cage to say anything. We do not always see eye-to-eye as he has the annoying habit to always try to see things from other people’s perspectives… I think though that fundamentally he agrees with me on the nature of the scientific method and I hope he will manage to bring this point across. Either way, here are some questions I think he should raise:

I. The Crusaders claim that a majority of published research findings are false positives. So far nobody has given a satisfying answer to the question how many false positives should we tolerate? 5%? 1%? Or zero? Is that realistic?

II. The Crusader types often complain about high-impact journals like Nature and Science because they value “novelty” and high risk findings. This supposedly hurts the scientific ideal of seeking the truth. But is that true? Don’t we need surprising, novel, paradigm-shifting findings to move science forward? Over the years of watching Sam do science I have developed a strong degree of cynicism. The scientific truth is never as simple and straightforward as the narrative presented in research papers – including those in most low-impact journals. Science publishing should be about communicating results and hypotheses even if they are wrong – because in the long run most inevitably will be.

III. We can expect there to be at least some discussion of preregistration at the debate. This will be interesting because Sophie has been one of the few outspoken critics of this idea. I have of course also written about this before and it’s a shame that I can’t participate directly in this debate. We would make one awesome dynamic duo of science ladies raging against preregistration! However much Sam will let me say about this though, putting all the naive debate about the potential benefit and problems of this practice aside, the thing that has bothered me most about it is that we have no compelling evidence for or against preregistration. What is more, there also seems to be little consensus on how can we  even establish whether preregistration works. To me these are fundamental concerns that we need to address now before this idea really takes off. Isn’t it ironic that given the topic of this debate, nobody seems to seriously talk about these points? Shouldn’t scientists value evidence above all else? Shouldn’t proponents of preregistration be expected to preregister the design of their revolutionary experiment?

IV. In addition to these rather lofty concerns, I also have a pragmatic question. Is the Crusaders’ behavior responsible? Even assuming that people do not engage in outright questionable research practices (and I remain unconvinced that they are as widespread as the Crusaders say), I think you can almost guarantee that most preregistered research findings will be lackluster compared to those published in the traditional model. In a world where the traditional model dominates, won’t that harm the junior researchers who are coerced to follow the rigid model? A quote I heard on the grapevine was “My supervisor wants me to preregister all my experiments and it’s destroying my career”. How do we address this problem?

As I said, Sam might not let me say these things and he doesn’t agree with me on many accounts. However, I hope there will be some debate around these questions. I will try to report back soon although Sam is demanding a lot of time for himself these days… Stay tuned.

Advertisements

9 thoughts on “Is Science broken?

  1. Dear Devil’s Neurscientist,

    I am sure you are aware of the Stapel case, in which a social psychologist made up data. I am wondering whether you would see anything “scientifically wrong” with that kind of behaviour?

    One could say that hypothesising and subsequently making up data nevertheless has lead to other scientists building up on Stapel’s work and hypotheses. One could say it still adheres to “Science publishing should be about communicating results and hypotheses even if they are wrong (…)”.

    Is there anything “scientifically wrong” with Stapel’s behaviour, and if so what exactly?

    Like

    • Sorry, are you kidding me? I should probably not dignify this obvious trolling with a response but then again I am representing the side of demons and monsters, so lucky for you I have a weakness for trolls. So just briefly some thoughts:

      If you had bothered to actually read my blog, you would know that I have written about the Stapel case (and the Schon case, the Hwang case, and I may even have mentioned the Obokata case – just in case you still think that psychology is somehow “special”). You’d also know that even the fiercest Crusaders acknowledge that the Crusade isn’t about stopping outright fraudsters. And you’d know that I have never argued that we cannot think of ways to improve the reliability of published research findings. All I am saying is that if you expect that every published research finding is the end of the story, you’re really missing the point.

      Anyway, please come back to discuss if you’re actually willing to read what I wrote and engage in meaningful discussion about the question I raise. But if you only want to continue this anonymous trolling with strawman arguments, then please go waste somebody else’s time.

      Like

      • Dear Devil’s Neuroscientist,

        I am not trolling. I am/ was genuinely interested in what is “scientifically wrong” about faking data in light of some of your statements. I am trying to understand, but apparently I am not doing a good job at that.

        Like

  2. You call me “one of the generals of the Crusade for True Science”. I have to say I take issue with this. Casting those who propose pre-registration or Registered Reports as one way of improving reproducibility as “crusaders” misrepresents our objectives and turns a discussion about science – something we all care about – into something judgmental, moralistic and self-righteous. Frankly it comes across as mudslinging.

    I have a pretty thick skin and usually stay silent when this happens (though I find it a little tiresome to be honest) but this kind of rhetoric risks toxifying these discussions. I’ve had junior researchers, and some women researchers, tell me that one reason they steer clear of these discussions is the risk of being attributed these kinds of labels.

    I’m sure that’s not your intention and you’re just trying to be humorous (or maybe you’ll respond that this is the “Devil” talking, not you — please don’t try that one), but in my opinion it’s not helpful and comes across as trolling.

    Like

    • I won’t make a detailed reply because the weekend is Sam’s time usually. Also I think we should first let you guys have the discussion on Tuesday. I hope there will be at least some answers to the questions I’m asking.

      Just briefly though, while I’m sorry that you feel offended by being called a “crusader” (which has been a theme on this blog since its inception), I think you are missing the point. If the whole movement felt less like a crusade to me and many others who are afraid to speak out against it for fear of being the next target of the open-science-let’s-replicate-anything-under-the-sun crowd, I wouldn’t call it such. You wouldn’t believe how many people have come to me expressing their support to what I’m doing here. Many of them are women, not that it matters. I’m arguing on behalf of all scientists who feel victimized by this movement, whether they are male, female, white, black, human or Martian.

      Of course, nobody has come to me personally. People come to Sam and express their views to him. To be fair, except for directly commenting on this blog this is the only way to communicate with me. However, it continues to baffle me how often people conflate the two of us. You just did it again. I’m not “trying” anything. If this still confuses you, please visit the About section on my blog and read the opening statement (first post).

      Honestly, if there is one thing that is broken about science then it is that scientists have lost the tradition of their philosopher ancestors to have an impassioned but detached argument.

      Like

    • To note; the OED definition of ‘crusader’ is ‘person who campaigns vigorously for political, social, or religious change; a campaigner’.

      Like

  3. Thank you for introducing me to this controversy and the concept of preregistration. As a clinician and former researcher (pursuing both clinical and benchtop investigation in the biology of trauma) I have also developed a healthy cynicism for what passes for science in our peer-reviewed journals. It seems to me, however, that there is a wide disparity in what qualifies as science. Spend a while thumbing through the surgical journals and you’ll laugh till you cry. I know I do and I spent over a decade cultivating the stuff. Preregistration won’t begin to address the quality issues that afflict the majority of articles published in our medical journals. Yet there continues to be great pressure on both author and publisher to keep this hamster wheel spinning. Is it the science that is broken or the way we define what passes for science at the moment?

    Like

    • Thanks for your comment. I think you’re hitting on a key issue here: in order for science to do well at self-correction or, in your words, to know what ‘passes for science’ we need better evaluation, greater scrutiny, and more discussion. All of these will only come about through better education. I don’t think any idealistic publication models are going to tackle the problems with bad science. The only thing that can do that is good scientists and incentives for them to actually do their jobs.

      Like

Comments are closed.