In response to my previous post, a young (my age judgment may be way off, I’m just going by the creative use of punctuation marks) mind by the name of ‘confused’ left a very insightful comment. I was in the process of writing up my response when I realized that the reply would work far better as another post because it explains my thinking very well, perhaps better than all the verbal diarrhea I have produced over the past few months.
For the sake of context, I will post the whole comment. First, the paragraph from my previous post that this person/demon/interdimensional entity commented on:
Devil’s Neuroscientist: “There is no ‘maybe’ about it. As I have argued in several of my posts, it is inherent to the scientific process. Science may get stuck in local minima and it may look like a random walk before converging on the truth – but given sufficient time and resources science will self-correct.”
And their response:
confused: “I have a huge problem with these statements. Who says science will self-correct?!?! Maybe certain false-positive findings will be left alone and no-one will investigate them any further. At that point you have an incorrect scientific record.
Also, saying “given sufficient time and resources science will self-correct” is a statement that is very easy to use to wipe all problems with current day science under the rug: nothing to see here, move along, move along… We scientists know what’s best, don’t you worry your pretty little head about it…”
I will address this whole comment here point by point.
“I have a huge problem with these statements. Who says science will self-correct?!?!”
I’ve answered this question many times before. Briefly, I’ve likened science to a model fitting algorithm. Algorithms may take a long time to converge on a sensible solution. In fact, they may even get stuck completely. In that situation, all that can help is to give it a push to a more informed place to search for the solution. This push may be because novel technologies provide new and/or better knowledge but they may instead simply come from the mind of a researcher who dares to think outside the box. The history of science is literally full of examples for either case.
This is what “inherent to the process” means. It is the sole function of science to self-correct because the whole point of science is to improve our understanding of the world. Yes, it may take a long time. But as long as the scientific spirit drives inquisitive minds to understand, it will happen eventually – provided we don’t get obliterated by an asteroid impact, a hypernova, or – what would be infinitely worse – by our own stupidity.
“Maybe certain false-positive findings will be left alone and no-one will investigate them any further.”
Undoubtedly this is the case. But is this a problem? First of all, I am not sure why I should care about findings that are not investigated any further. I don’t know about you but to me this sounds like nobody else cares about them either. This may be because everybody feels like they are spurious or perhaps because they simply just ain’t very important.
However, let me indulge you for a moment and assume that somebody actually does care about the finding, possibly someone who is not a scientist. In the worst possible case, they could be a politician. By all that is sacred, someone should look into it then and find out what’s going on! But in order to do so, you need to have a good theory, or at least a viable alternative hypothesis, not the null. If you are convinced something isn’t true, show me why. It does not suffice to herald each direct non-replication as evidence that the original finding was a false positive because in reality these kind of discussions are like this.
“At that point you have an incorrect scientific record.”
Honestly, this statement summarizes just about everything that is wrong with the Crusade for True Science. The problem is not that there may be mistakes in the scientific record but the megalomaniac delusion that there is such a thing as a “correct” scientific record. Science is always wrong. It’s inherent to the process to be wrong and to gradually self-correct.
As I said above, the scientific record is full of false positives because this is how it works. Fortunately, I think in the vast majority of false positives in the record are completely benign. They will either be corrected or they will pass into oblivion. The false theories that I worry about are the ones that most sane scientists already reject anyway: creationism, climate change denial, the anti-vaccine movement, Susan Greenfield’s ideas about the modern world, or (to stay with present events) the notion that you can “walk off your Parkinson’s.” Ideas like these are extremely dangerous and they have true potential to steer public policy in a very bad direction.
In contrast, I don’t really care very much whether priming somebody with the concept of a professor makes them perform better at answering trivia questions. I personally doubt it and I suspect simpler explanations (including that it could be completely spurious) but the way to prove that is to disprove that the original result could have occurred, not to show that you are incapable of reproducing it. If that sounds a lot more difficult than to churn out one failed replication after another, then that’s because it is!
“Also, saying “given sufficient time and resources science will self-correct” is a statement that is very easy to use to wipe all problems with current day science under the rug: nothing to see here, move along, move along… “
Nothing is being swept under any rugs here. For one thing, I remain unconvinced by the so-called evidence that current day science has a massive problem. The Schoens and Stapels don’t count. There have always been scientific frauds and we really shouldn’t even be talking about the fraudsters. So, ahm, sorry for bringing them up.
The real issue that has all the Crusaders riled up so much is that the current situation apparently generates a far greater proportion of false positives than is necessary. There is a nugget of truth to this notion but I think the anxiety is misplaced. I am all in favor of measures to reduce the propensity of false positives through better statistical and experimental practices. More importantly, we should reward good science rather than sensational science.
This is why the Crusaders promote preregistration – however, I don’t think this is going to help. It is only ever going to cure the symptom but not the cause of the problem. The underlying cause, the actual sickness that has infected modern science, is the misguided idea that hypothesis-driven research is somehow better than exploratory science. And sadly, this sickness plagues the Crusaders more than anyone. Instead of preregistration, which – despite all the protestations to the contrary – implicitly places greater value on “purely confirmatory research” than on exploratory science, what we should do is reward good exploration. If we did that instead of insisting that grant proposals list clear hypotheses, “anticipating” results in our introduction sections, and harping on about preregistered methods, and if we were also more honest about the fact that scientific findings and hypotheses are usually never really fully true and we did a better job communicating this to the public, then current day science probably wouldn’t have any of these problems.
“We scientists know what’s best, don’t you worry your pretty little head about it…”
Who’s saying this? The whole point I have been arguing is that scientists don’t know what’s best. What I find so exhilarating about being a scientist is that this is a profession, quite possibly the only profession, in which you can be completely honest about the fact that you don’t really know anything. We are not in the business of knowing but in asking better questions.
Please do worry your pretty little head! That’s another great thing about being a scientist. We don’t live in ivory towers. Given the opportunity, anyone can be a scientist. I might take your opinion on quantum mechanics more seriously if you have the education and expertise to back it up, but in the end that is a prior. A spark of genius can come from anywhere.
What should we do?
If you have a doubt in some reported finding, go and ask questions about it. Think about alternative, simpler explanations for it. Design and conduct experiments to test this explanation. Then, report your results to the world and discuss the merits and flaws of your studies. Refine your ideas and designs and repeat the process over and over. In the end there will be a body of evidence. It will either convince you that your doubt was right or it won’t. More importantly, it may also be seen by many others and they can form their own opinions. They might come up with their own theories and with experiments to test them.
Doesn’t this sound like a perfect solution to our problems? If only there were a name for this process…