Confusions of Grandeur

In response to my previous post, a young (my age judgment may be way off, I’m just going by the creative use of punctuation marks) mind by the name of ‘confused’ left a very insightful comment. I was in the process of writing up my response when I realized that the reply would work far better as another post because it explains my thinking very well, perhaps better than all the verbal diarrhea I have produced over the past few months.

For the sake of context, I will post the whole comment. First, the paragraph from my previous post that this person/demon/interdimensional entity commented on:

Devil’s Neuroscientist: “There is no ‘maybe’ about it. As I have argued in several of my posts, it is inherent to the scientific process. Science may get stuck in local minima and it may look like a random walk before converging on the truth – but given sufficient time and resources science will self-correct.”

And their response:

confused: “I have a huge problem with these statements. Who says science will self-correct?!?! Maybe certain false-positive findings will be left alone and no-one will investigate them any further. At that point you have an incorrect scientific record.

Also, saying “given sufficient time and resources science will self-correct” is a statement that is very easy to use to wipe all problems with current day science under the rug: nothing to see here, move along, move along… We scientists know what’s best, don’t you worry your pretty little head about it…”

I will address this whole comment here point by point.

“I have a huge problem with these statements. Who says science will self-correct?!?!”

I’ve answered this question many times before. Briefly, I’ve likened science to a model fitting algorithm. Algorithms may take a long time to converge on a sensible solution. In fact, they may even get stuck completely. In that situation, all that can help is to give it a push to a more informed place to search for the solution. This push may be because novel technologies provide new and/or better knowledge but they may instead simply come from the mind of a researcher who dares to think outside the box. The history of science is literally full of examples for either case.

This is what “inherent to the process” means. It is the sole function of science to self-correct because the whole point of science is to improve our understanding of the world. Yes, it may take a long time. But as long as the scientific spirit drives inquisitive minds to understand, it will happen eventually – provided we don’t get obliterated by an asteroid impact, a hypernova, or – what would be infinitely worse – by our own stupidity.

“Maybe certain false-positive findings will be left alone and no-one will investigate them any further.”

Undoubtedly this is the case. But is this a problem? First of all, I am not sure why I should care about findings that are not investigated any further. I don’t know about you but to me this sounds like nobody else cares about them either. This may be because everybody feels like they are spurious or perhaps because they simply just ain’t very important.

However, let me indulge you for a moment and assume that somebody actually does care about the finding, possibly someone who is not a scientist. In the worst possible case, they could be a politician. By all that is sacred, someone should look into it then and find out what’s going on! But in order to do so, you need to have a good theory, or at least a viable alternative hypothesis, not the null. If you are convinced something isn’t true, show me why. It does not suffice to herald each direct non-replication as evidence that the original finding was a false positive because in reality these kind of discussions are like this.

“At that point you have an incorrect scientific record.”

Honestly, this statement summarizes just about everything that is wrong with the Crusade for True Science. The problem is not that there may be mistakes in the scientific record but the megalomaniac delusion that there is such a thing as a “correct” scientific record. Science is always wrong. It’s inherent to the process to be wrong and to gradually self-correct.

As I said above, the scientific record is full of false positives because this is how it works. Fortunately, I think in the vast majority of false positives in the record are completely benign. They will either be corrected or they will pass into oblivion. The false theories that I worry about are the ones that most sane scientists already reject anyway: creationism, climate change denial, the anti-vaccine movement, Susan Greenfield’s ideas about the modern world, or (to stay with present events) the notion that you can “walk off your Parkinson’s.” Ideas like these are extremely dangerous and they have true potential to steer public policy in a very bad direction.

In contrast, I don’t really care very much whether priming somebody with the concept of a professor makes them perform better at answering trivia questions. I personally doubt it and I suspect simpler explanations (including that it could be completely spurious) but the way to prove that is to disprove that the original result could have occurred, not to show that you are incapable of reproducing it. If that sounds a lot more difficult than to churn out one failed replication after another, then that’s because it is!

“Also, saying “given sufficient time and resources science will self-correct” is a statement that is very easy to use to wipe all problems with current day science under the rug: nothing to see here, move along, move along… “

Nothing is being swept under any rugs here. For one thing, I remain unconvinced by the so-called evidence that current day science has a massive problem. The Schoens and Stapels don’t count. There have always been scientific frauds and we really shouldn’t even be talking about the fraudsters. So, ahm, sorry for bringing them up.

The real issue that has all the Crusaders riled up so much is that the current situation apparently generates a far greater proportion of false positives than is necessary. There is a nugget of truth to this notion but I think the anxiety is misplaced. I am all in favor of measures to reduce the propensity of false positives through better statistical and experimental practices. More importantly, we should reward good science rather than sensational science.

This is why the Crusaders promote preregistration – however, I don’t think this is going to help. It is only ever going to cure the symptom but not the cause of the problem. The underlying cause, the actual sickness that has infected modern science, is the misguided idea that hypothesis-driven research is somehow better than exploratory science. And sadly, this sickness plagues the Crusaders more than anyone. Instead of preregistration, which – despite all the protestations to the contrary – implicitly places greater value on “purely confirmatory research” than on exploratory science, what we should do is reward good exploration. If we did that instead of insisting that grant proposals list clear hypotheses, “anticipating” results in our introduction sections, and harping on about preregistered methods, and if we were also more honest about the fact that scientific findings and hypotheses are usually never really fully true and we did a better job communicating this to the public, then current day science probably wouldn’t have any of these problems.

“We scientists know what’s best, don’t you worry your pretty little head about it…”

Who’s saying this? The whole point I have been arguing is that scientists don’t know what’s best. What I find so exhilarating about being a scientist is that this is a profession, quite possibly the only profession, in which you can be completely honest about the fact that you don’t really know anything. We are not in the business of knowing but in asking better questions.

Please do worry your pretty little head! That’s another great thing about being a scientist. We don’t live in ivory towers. Given the opportunity, anyone can be a scientist. I might take your opinion on quantum mechanics more seriously if you have the education and expertise to back it up, but in the end that is a prior. A spark of genius can come from anywhere.

What should we do?

If you have a doubt in some reported finding, go and ask questions about it. Think about alternative, simpler explanations for it. Design and conduct experiments to test this explanation. Then, report your results to the world and discuss the merits and flaws of your studies. Refine your ideas and designs and repeat the process over and over. In the end there will be a body of evidence. It will either convince you that your doubt was right or it won’t. More importantly, it may also be seen by many others and they can form their own opinions. They might come up with their own theories and with experiments to test them.

Doesn’t this sound like a perfect solution to our problems? If only there were a name for this process…

In the words of the great poet and philosopher Bimt Lizkip, failed direct replications in psychology research are just “He Said She Said Bulls**t”

How science works

My previous post discussed the myths surrounding the “replication crisis” in psychology/neuroscience research. As usual, it became way too long and I didn’t even cover several additional points I wanted to mention. I will leave most of these for a later post in which I will speculate about why failed replications, papers about incorrect/questionable procedures, and other actions by the Holy Warriors for Research Truth cause such a lot of bad blood. I will try to be quick in that one or split it up into parts. Before I can get around to this though, let me briefly (and I am really trying this time!) have a short intermission with practical examples of the largely theoretical and philosophical arguments I made in previous posts.

Science is self-correcting

I’ve said it before but it deserves saying again. Science self-corrects, no matter how much the Crusaders want to whine and claim that this is a myth. Self-correction isn’t always very fast. It can take decades or even centuries. When thoroughly incorrect theories take root, it can take some extraordinary effort to uproot them again. However, given the pursuit of truth and free expression continues unhindered and as long as research has sufficient resources and opportunities, a correction will eventually happen. In many ways self-correction is like evolution, climate change, or plate tectonics. You rarely see direct evidence for these slow changes by looking at single time points but the trends are clearly visible. Fortunately though, scientific self-correction is typically faster than these things.

Take the discovery last year of gravitational waves supporting the Big Bang theory of the origin of the universe. It was widely reported in the news media as a earth- (well, space-) shattering finding that would greatly advance our understanding of our place in the cosmos. It resulted in news articles, media reports, and endless social media shares of videos and articles about this and the Daily Mail immediately making a complete fool of itself as usual by revealing just how little they know about the scientific process. It sparked some discussion about sexism in science journalism because initial reports ignored one of the female scientists who inspired the theory. (I am too lazy to provide any links to these things. If you’re on this blog I assume you’ve mastered the art of google).

However, in the months that followed critical voices began to be heard suggesting that this discovery may have been a fluke. Instead of gravitational waves revealing the fabric of the universe itself, it appears that these measurements were contaminated by signals stemming from cosmic dust. Just the other day, Nature published a story suggesting that this alternative, simpler explanation now looks to be correct. I am not the Devil’s Astrophysicist so I can’t talk about the specifics of this research but it certainly sounds compelling from my laywoman’s  perspective: even without in-depth understanding of the research I can tell that the evidence suggests a simpler explanation for the initial findings and by Occam’s Razor alone I would be encouraged to accept this one as more probable.

They apparently also have orientation columns in space

I don’t see any problem with this and I am pretty sure neither do most scientists. This is just self-correction in action. If anything about this story has been problematic, it is the fact that the initial discovery was so widely publicized before it was validated. I don’t really know where I stand on this. On the one hand, this has been unfortunate because it could greatly confuse public opinion about this research. The same applies to the now “debunked” finding of arsenic life forms reported a few years ago. A spectacular, possibly paradigm-shifting discovery was reported widely only to be disproved at a later point. I could see that this sort of perceived instability in scientific discoveries could undermine the public’s trust in science with potentially devastating consequences. In a world where climate change deniers, “anti-vaxers”, and creationists are given an undeservedly strong platform to spread their views (no linking to those either for that very reason), we need to be very careful how science is communicated. A lack of faith in scientific research could do a lot of harm.

However, in my mind this doesn’t mean we should suppress media reports of new, possibly not-yet-validated discoveries. For one thing, in a free society we can’t forbid people from talking about their research. I also believe that science should be a leading example of democratic ideals and transparency because good science is based on it. Instead of strong regulation and rigid structures science flourishes when authority is questioned.

“[Science] connects us with our origins, and it too has its rituals and its commandments. Its only sacred truth is that there are no sacred truths. All assumptions must be critically examined. Arguments from authority are worthless.”

Carl Sagan – Cosmos, Episode 13 “Who speaks for Earth?”

More importantly, I believe as scientists we actually want to communicate the enthusiasm and joy we experience when we reveal secrets of the world – even if they are untrue. Rather than curtailing when and how we communicate our discoveries to the public, we should do a better job at communicating how science works in practice. By all means, we should improve our press releases to minimize the sweeping generalizations and unfounded speculations by the news media about the implications of a new finding. When communicating sensational results to the public, we should certainly convey our excitement. However, we should also always remind them (and ourselves) that any new discovery is never the last word on an issue but the first.

In my view helping the public understand that scientific knowledge is ever evolving and that “boffins” didn’t just “prove” things will in fact strengthen the public’s trust in science. The reason I, as a neuroscientist without any deeper understanding of atmospheric physics, believe that man-made climate change is a problem is that an overwhelming majority of climate scientists believe the evidence supports this hypothesis. I know what scientists are like. They disagree all the damn time about the smallest issues. If there is an overwhelming consensus on any point it is past time we listen!

The Psychoplication Crisis

After having talked about gravitational waves and arsenic bacteria, you can bet some smartass will come out of the woodwork and tell you that these are the “hard” natural sciences but the same does not apply to “soft” science like psychology or cognitive neuroscience. Within psychology/cognitive neuroscience the same snobbery exists between experimental psychology and social cognition researchers. Perhaps that is somewhat understandable as some of the first perception researchers were in fact physicists (this is the origin of the term psychophysics). However, I think this sort of segregation is delusional and the self-satisfied belief that your own research area is somehow a “harder” science can be quite dangerous because it makes it all that much easier to mislead you into thinking that your own research is not susceptible to these problems.

I don’t think the problems that are currently discussed in psychology and neuroscience are in any way specific to or even particularly pervasive in our field. Some concepts and theories are simply more established in physics than they are in psychology. We certainly don’t need to do any experiments to prove gravity or even to confirm many of the laws that seem to govern it. The same cannot be said about social psychology where people still question whether things like unconscious priming of complex behaviors exist in the first place. But the fact that sensational findings are hyped up by media reports only to be disproved in spectacular failures to replicate is evidently just as common in physics and microbiology. And even shocking revelations of scientific fraud have happened in those fields (see also here for a less clear-cut case – I’ll probably discuss this more next time).

One way in which the culture in physics research seems to differ from our field is that it is far more common to upload manuscripts to public repositories prior to peer review. This allows a wider, more public discussion of the purported discoveries which is certainly good for the scientific process. However, it also results in broader media coverage before the research has even passed the most basic peer review stage and this can in fact exacerbate the problems with unverified findings. The still rare cases of this approach in psychology research suffer from the same problem. Just look at Daryl Bem’s meta-analysis of “precognition” experiments which is available on such a repository even though it has not been formally published by a peer reviewed journal. Unlike his earlier findings, which were also widely discussed before they were published officially, this analysis hasn’t been picked up by many news outlets. However, it easily could have been, just as was the case for the earlier examples from physics and microbiology research. Another famous parapsychologist also posted it on his blog of as evidence that “the critics need to rethink their position” – a somewhat premature conclusion if you ask me.

In theory, I don’t see a problem with having new, un-reviewed manuscripts publicly available provided that the surrounding peer discussion is also visible. The problem is that this is not always the case. Certainly the news coverage of the peer discussion usually doesn’t match the media circus surrounding the original finding. Taking Bem’s meta-analysis as an example, it obviously has been under closed peer review for at least the better part of a year now. I would love to actually see this peer review discussion as it is developing but – with one exception (to my knowledge) – so far this review has been for the eyes of the peer reviewers only.