In response to my previous post, a young (my age judgment may be way off, I’m just going by the creative use of punctuation marks) mind by the name of ‘confused’ left a very insightful comment. I was in the process of writing up my response when I realized that the reply would work far better as another post because it explains my thinking very well, perhaps better than all the verbal diarrhea I have produced over the past few months.
For the sake of context, I will post the whole comment. First, the paragraph from my previous post that this person/demon/interdimensional entity commented on:
Devil’s Neuroscientist: “There is no ‘maybe’ about it. As I have argued in several of my posts, it is inherent to the scientific process. Science may get stuck in local minima and it may look like a random walk before converging on the truth – but given sufficient time and resources science will self-correct.”
And their response:
confused: “I have a huge problem with these statements. Who says science will self-correct?!?! Maybe certain false-positive findings will be left alone and no-one will investigate them any further. At that point you have an incorrect scientific record.
Also, saying “given sufficient time and resources science will self-correct” is a statement that is very easy to use to wipe all problems with current day science under the rug: nothing to see here, move along, move along… We scientists know what’s best, don’t you worry your pretty little head about it…”
I will address this whole comment here point by point.
“I have a huge problem with these statements. Who says science will self-correct?!?!”
I’ve answered this question many times before. Briefly, I’ve likened science to a model fitting algorithm. Algorithms may take a long time to converge on a sensible solution. In fact, they may even get stuck completely. In that situation, all that can help is to give it a push to a more informed place to search for the solution. This push may be because novel technologies provide new and/or better knowledge but they may instead simply come from the mind of a researcher who dares to think outside the box. The history of science is literally full of examples for either case.
This is what “inherent to the process” means. It is the sole function of science to self-correct because the whole point of science is to improve our understanding of the world. Yes, it may take a long time. But as long as the scientific spirit drives inquisitive minds to understand, it will happen eventually – provided we don’t get obliterated by an asteroid impact, a hypernova, or – what would be infinitely worse – by our own stupidity.
“Maybe certain false-positive findings will be left alone and no-one will investigate them any further.”
Undoubtedly this is the case. But is this a problem? First of all, I am not sure why I should care about findings that are not investigated any further. I don’t know about you but to me this sounds like nobody else cares about them either. This may be because everybody feels like they are spurious or perhaps because they simply just ain’t very important.
However, let me indulge you for a moment and assume that somebody actually does care about the finding, possibly someone who is not a scientist. In the worst possible case, they could be a politician. By all that is sacred, someone should look into it then and find out what’s going on! But in order to do so, you need to have a good theory, or at least a viable alternative hypothesis, not the null. If you are convinced something isn’t true, show me why. It does not suffice to herald each direct non-replication as evidence that the original finding was a false positive because in reality these kind of discussions are like this.
“At that point you have an incorrect scientific record.”
Honestly, this statement summarizes just about everything that is wrong with the Crusade for True Science. The problem is not that there may be mistakes in the scientific record but the megalomaniac delusion that there is such a thing as a “correct” scientific record. Science is always wrong. It’s inherent to the process to be wrong and to gradually self-correct.
As I said above, the scientific record is full of false positives because this is how it works. Fortunately, I think in the vast majority of false positives in the record are completely benign. They will either be corrected or they will pass into oblivion. The false theories that I worry about are the ones that most sane scientists already reject anyway: creationism, climate change denial, the anti-vaccine movement, Susan Greenfield’s ideas about the modern world, or (to stay with present events) the notion that you can “walk off your Parkinson’s.” Ideas like these are extremely dangerous and they have true potential to steer public policy in a very bad direction.
In contrast, I don’t really care very much whether priming somebody with the concept of a professor makes them perform better at answering trivia questions. I personally doubt it and I suspect simpler explanations (including that it could be completely spurious) but the way to prove that is to disprove that the original result could have occurred, not to show that you are incapable of reproducing it. If that sounds a lot more difficult than to churn out one failed replication after another, then that’s because it is!
“Also, saying “given sufficient time and resources science will self-correct” is a statement that is very easy to use to wipe all problems with current day science under the rug: nothing to see here, move along, move along… “
Nothing is being swept under any rugs here. For one thing, I remain unconvinced by the so-called evidence that current day science has a massive problem. The Schoens and Stapels don’t count. There have always been scientific frauds and we really shouldn’t even be talking about the fraudsters. So, ahm, sorry for bringing them up.
The real issue that has all the Crusaders riled up so much is that the current situation apparently generates a far greater proportion of false positives than is necessary. There is a nugget of truth to this notion but I think the anxiety is misplaced. I am all in favor of measures to reduce the propensity of false positives through better statistical and experimental practices. More importantly, we should reward good science rather than sensational science.
This is why the Crusaders promote preregistration – however, I don’t think this is going to help. It is only ever going to cure the symptom but not the cause of the problem. The underlying cause, the actual sickness that has infected modern science, is the misguided idea that hypothesis-driven research is somehow better than exploratory science. And sadly, this sickness plagues the Crusaders more than anyone. Instead of preregistration, which – despite all the protestations to the contrary – implicitly places greater value on “purely confirmatory research” than on exploratory science, what we should do is reward good exploration. If we did that instead of insisting that grant proposals list clear hypotheses, “anticipating” results in our introduction sections, and harping on about preregistered methods, and if we were also more honest about the fact that scientific findings and hypotheses are usually never really fully true and we did a better job communicating this to the public, then current day science probably wouldn’t have any of these problems.
“We scientists know what’s best, don’t you worry your pretty little head about it…”
Who’s saying this? The whole point I have been arguing is that scientists don’t know what’s best. What I find so exhilarating about being a scientist is that this is a profession, quite possibly the only profession, in which you can be completely honest about the fact that you don’t really know anything. We are not in the business of knowing but in asking better questions.
Please do worry your pretty little head! That’s another great thing about being a scientist. We don’t live in ivory towers. Given the opportunity, anyone can be a scientist. I might take your opinion on quantum mechanics more seriously if you have the education and expertise to back it up, but in the end that is a prior. A spark of genius can come from anywhere.
What should we do?
If you have a doubt in some reported finding, go and ask questions about it. Think about alternative, simpler explanations for it. Design and conduct experiments to test this explanation. Then, report your results to the world and discuss the merits and flaws of your studies. Refine your ideas and designs and repeat the process over and over. In the end there will be a body of evidence. It will either convince you that your doubt was right or it won’t. More importantly, it may also be seen by many others and they can form their own opinions. They might come up with their own theories and with experiments to test them.
Doesn’t this sound like a perfect solution to our problems? If only there were a name for this process…
In the words of the great poet and philosopher Bimt Lizkip, failed direct replications in psychology research are just “He Said She Said Bulls**t”
The Devil's Neuroscientist 
Good post and the discussion following it.
Keep up with the blog, dude!
LikeLike
Thanks for the support! Although the Devil’s Neuroscientist is a lady, not a ‘dude’…
LikeLike
Sorry for that, my lady 😉
In any case, I look forward to more posts on your blog 🙂
LikeLike
Thank you for your reply!
From the dictionary:
Science
1.
a branch of knowledge or study dealing with a body of facts or truths systematically arranged and showing the operation of general laws
1) “Maybe certain false-positive findings will be left alone and no-one will investigate them any further.”
Undoubtedly this is the case. But is this a problem? First of all, I am not sure why I should care about findings that are not investigated any further. I don’t know about you but to me this sounds like nobody else cares about them either. This may be because everybody feels like they are spurious or perhaps because they simply just ain’t very important.”
When nobody cares about a result is not necessarily a good thing or an indication as to whether the effect is true. Perhaps the result is tremendously important and relevant and true, but no-one pays any attention to it (perhaps it’s not been written by some flashy social-psychologist super star like Stapel).
In fact, I have often wondered whether in today’s science attention for a certain effect/ theory is inversely related to its importance, relevance, and truth value (e.g. “priming”, “embodied cognition”).
2) ““Also, saying “given sufficient time and resources science will self-correct” is a statement that is very easy to use to wipe all problems with current day science under the rug: nothing to see here, move along, move along… “
Nothing is being swept under any rugs here. For one thing, I remain unconvinced by the so-called evidence that current day science has a massive problem. The Schoens and Stapels don’t count. There have always been scientific frauds and we really shouldn’t even be talking about the fraudsters. So, ahm, sorry for bringing them up.”
I don’t understand how you remain inconvinceed by the “so-called evidence that current day science has a massive problem”. How can you read this (http://pss.sagepub.com/content/22/11/1359.short?rss=1&ssource=mfc) and not laugh out loud and completely lose your faith in the majority of psychological studies.
Stapel, in my opinion, does count very heavily. Not because he was a fraud, but because of the report on his fraud that states things like:
“In the case of the fraud committed by Mr Stapel, the critical function of science has failed on all levels. Fundamental principles of scientific method have been ignored, or set aside as irrelevant.” (p. 54)
(https://www.commissielevelt.nl/wp-content/uploads_per_blog/commissielevelt/2013/01/finalreportLevelt1.pdf)
3) “We scientists know what’s best, don’t you worry your pretty little head about it…”
Who’s saying this? The whole point I have been arguing is that scientists don’t know what’s best. ”
Good 🙂 Let’s stop then with the university’s PR machines that spit out nonsense. Let’s stop reporting on science altogether, perhaps only some meta-analysis results with carefully framed conclusions (if any). Let’s stop awarding individual awards to scientists. Let’s stop financing more underpowered random noise studies: if the scientists are so eager to share these “important” results, why not have them write a diary instead of polluting the scientific record.
4) “In the words of the great poet and philosopher Bimt Lizkip, failed direct replications in psychology research are just “He Said She Said Bulls**t”
And that’s why psychology will never amount to anything useful in my opinion. For the great majority of it, it’s just like an opinion because people in it apparently hold different standards to the work, and fail to adjust their opinion based on new evidence (http://www.nature.com/news/unconscious-thought-not-so-smart-after-all-1.16801?WT.mc_id=TWT_NatureNews)
5) p.s. kudos for limp bizkit.
LikeLike
Thanks for your comment. I’ll try to be brief (but wasn’t ;):
Perhaps the result is tremendously important and relevant and true, but no-one pays any attention to it
Sure. That is why we should keep our eyes out for those pearls of wisdom that others are missing. We should improve our ability to find those findings. But in the end, that’s just life. The unrealistic post-crusade utopia will not fix that problem either. It’s up to us to look for important findings and to convince our colleagues why they matter. And then we must scrutinize and test them so they are either falsified or confirmed.
(perhaps it’s not been written by some flashy social-psychologist super star like Stapel).
After reading his book I’m honestly unsure that this is true. I certainly never heard his name until he became famous for being a fraudster.
But I guess the general point still stands. There are flashy super star scientists out there. Is that different from any other aspect of life though? How many politicians are elected not for the sensible policies and leadership skills but their charisma and erroneous assumptions about how they saved the economy?
If you suspect that some part of science is all showmanship without real substance, then go ahead and show me the evidence for that. That’s what scientists are supposed to do. Don’t just sit around whining about how nobody pays attention to you. It may be hard, it may take until that showman scientist dies or retires – but in the end strong evidence will trump celebrity.
In fact, I have often wondered whether in today’s science attention for a certain effect/ theory is inversely related to its importance, relevance, and truth value (e.g. “priming”, “embodied cognition”).
You may be right about some of those, but again, show me the evidence. Don’t just sit around “wondering”. Skepticism is good, but without evidence skepticism is not a useful contribution to our knowledge.
I don’t understand how you remain inconvinceed by the “so-called evidence that current day science has a massive problem”. How can you read this (http://pss.sagepub.com/content/22/11/1359.short?rss=1&ssource=mfc) and not laugh out loud and completely lose your faith in the majority of psychological studies.
For one thing, I am a natural skeptic and I don’t believe their claims about the severity of the problem. Looking through Sam’s eyes I have witnessed some individuals engaging in some serious questionable research practices. But I have also witnessed some shining examples of the opposite. In addition to that, while some questionable research practices really contaminate the literature, many effects may still be real even if they are inflated by the literature. The way to test research findings is not to sit around arguing about preregistration and p-curves, but to test them empirically.
“In the case of the fraud committed by Mr Stapel, the critical function of science has failed on all levels. Fundamental principles of scientific method have been ignored, or set aside as irrelevant.”
In all honesty, when reading the Stapel report I felt a sense of hubris in all that. Then again, some of the practices they claimed were widespread were indeed shocking and his book confirms some serious misunderstanding of what science should be about. This must change. That mainly involves educating people and rewarding good science.
Good 🙂 Let’s stop then with the university’s PR machines that spit out nonsense.
I agree. But you can’t stop reporting on science nor should we. Instead we should report how science really works, that new findings are never the last word on an issue but usually the first. We should emphasize that skepticism is necessary for healthy science.
Let’s stop awarding individual awards to scientists.
I think that’s unrealistic. There could be more communist models of funding science, but I am skeptical of communism. It is good for people to be rewarded and reaping in funding. What we should do is giving individual awards to scientists for the right reasons.
Let’s stop financing more underpowered random noise studies:
Sure, let’s stop underpowered research. Will you come with me to petition the government for more research funding?
if the scientists are so eager to share these “important” results, why not have them write a diary instead of polluting the scientific record.
Or alternatively we could just understand that this is what we are already doing. Individual flashy journal papers are just a diary of science too. We need to remember that. Don’t blame other scientists for “polluting” the scientific record. You are the one clouding your own judgment if you expect all (or most) published results to be the whole truth and nothing but the truth.
I think the scientific record should be full of results and ideas that turn out to be untrue. Reading the scientific record, the diary entries of “eager” scientists, is what gives us the impetus to ask more probing questions. I don’t want a literature full of boring meta-analyses. It will just end with Sam writing more commentaries about parapsychology research and this, my friend, is something I am working very hard to avoid.
And that’s why psychology will never amount to anything useful in my opinion. For the great majority of it, it’s just like an opinion because people in it apparently hold different standards to the work, and fail to adjust their opinion based on new evidence
See, and this is where I think you’re simply wrong. Your view is similarly misguided as saying that because you haven’t personally seen any new species emerge is proof that there is no evolution of species. You are basing an interpretation of a gradual process on a single, uninformative time point.
The fact that we’re having this discussion is disproving what you said. The fact that people question the results of social priming or whatnot, is evidence that psychology is very much subject to changing opinions. This right here is self-correction.
LikeLike
1) *”(perhaps it’s not been written by some flashy social-psychologist super star like Stapel).
-After reading his book I’m honestly unsure that this is true. I certainly never heard his name until he became famous for being a fraudster.”
*He was at least “flashy” enough to have received the “Society of Experimental Social Psychology” Career Trajectory Award in 2009, which has since been retracted as a result of his fraud (http://www.sesp.org/trajectory.htm). Looking over the other names, I know the work of only 2 of the others whose work coincidentally I both do not trust whatsoever. Another reason why individual awards should not be handed out in science i.m.o.
2) *”Let’s stop financing more underpowered random noise studies:
-Sure, let’s stop underpowered research. Will you come with me to petition the government for more research funding? ”
*I never understood this reasoning. Why not perform less studies that are more powerful? What’s so hard about doing this?
3) *”if the scientists are so eager to share these “important” results, why not have them write a diary instead of polluting the scientific record.
-Or alternatively we could just understand that this is what we are already doing. Individual flashy journal papers are just a diary of science too. We need to remember that. Don’t blame other scientists for “polluting” the scientific record. ”
I think there needs/ should be/ is a difference in “science” and “a diary of ‘science'”. I agree with you that no single study holds the whole truth, but looking at my earlier written up definition of science it should be about “facts” and “truths”. Reasoning from that point onwards, science should not be about “leading to more questions”. Sure, that may be a by-product of doing science, but that’s not what science is about if I understood, and used the definition of science correctly.
4) -“I don’t want a literature full of boring meta-analyses”
*How about a literature full of findings you can at leas somewhat trust. You know “facts” and “truths”. Imagine that you wouldn’t have to spend study after study trying to replicate someone’s false-positive. Imagine all highly powered, pre-registered studies that you can trust more than most of the stuff that is has been published over the last few decades. People can still do all the “exploratory work” they ever dreamed of, but they now merely follow that up with pre-registered, highly powered work which then tests this exploratory work so you can have greater confidence in what you finally read in the paper. I think that would all be great !
I thought scientists were “curious” and want to “try new things”, so why won’t they try the above scenario. They can then even investigate whether this leads to more robust results and stuff like that. Why is it that scientists are only “curious” and want to “try new things” when it comes to performing underpowered, random noise studies following the latest fad, and not want to explore important things like how to better science altogether !?!? I just don’t understand…
5) -“This right here is self-correction.”
Is it? Did we come closer to the truth? Or have we just engaged in some he said, she said, b@llsh#t ?
LikeLike
Looking over the other names, I know the work of only 2 of the others whose work coincidentally I both do not trust whatsoever
As I said, mistrust all you like – but if you want to convince anyone (including yourself!) that this mistrust is justified, provide some evidence. And no, failing to replicate somebody’s work once or twice is insufficient evidence. That’s just he-said-she-said-bulls**t (henceforth abbreviated as HSSSBS). A conclusive piece of evidence demonstrating why they are wrong in the first place means more than a million replications.
Regarding this list, to be perfectly honest I don’t know anyone there except for the one who looks like Ned Stark. Perhaps that’s because I’m not a social psychologist but if so then this puts the whole superstar celebrity claims into perspective.
I never understood this reasoning. Why not perform less studies that are more powerful? What’s so hard about doing this?
Because I am a scientist, I am naturally curious and inquisitive, and I think more science is better. But sure, there is a tradeoff. I never advocated that we should do more low-powered studies but I will always advocate doing more studies. I think science is better served by producing more results, which in turn lead to more hypotheses and more questions. If we were so conservative that we only have a handful highly trustworthy findings we would still be in the dark ages.
Reasoning from that point onwards, science should not be about “leading to more questions”. Sure, that may be a by-product of doing science, but that’s not what science is about if I understood, and used the definition of science correctly.
I am not sure dictionary definitions of ‘science’ help us very far. Science or research or whatever the hell you want to call it is the quest to better understand the universe. In the long run it is about asking some very fundamental questions.
I agree with you that no single study holds the whole truth, but looking at my earlier written up definition of science it should be about “facts” and “truths”.
I think you’re missing the point here. Science is the pursuit of facts and truths, which inevitably leads to asking more questions. If you want the final word on any issue, I’m afraid you’re in the wrong profession. In science ideas only matter if they can be falsified, and facts only count if they stand up to scrutiny. This entails asking more questions.
Imagine that you wouldn’t have to spend study after study trying to replicate someone’s false-positive.
But replication is essential to science! No result is truly trustworthy until it has been replicated. You’re basically expecting to have your cake and eat it too.
Would I like to see less false positives? Sure – although I don’t really know how many false positives there are because nobody can show me any actual hard evidence for that. All they talk about is posthoc power calculations and failed direct replications that lead us nowhere.
Imagine all highly powered, pre-registered studies that you can trust more than most of the stuff that is has been published over the last few decades.
What is it about pre-registration that makes a result trustworthy? I am a scientist and that makes me a skeptic. I am not convinced of grand ideas without hard evidence. This unfounded faith the crusaders have in pre-registration is potentially quite dangerous in my opinion.
People can still do all the “exploratory work” they ever dreamed of, but they now merely follow that up with pre-registered, highly powered work which then tests this exploratory work so you can have greater confidence in what you finally read in the paper. I think that would all be great!
Actually I don’t entirely disagree. As I said, I think the true problem is that exploratory research is undervalued. That is why people pretend to have hypothesized their findings when in truth they didn’t (or at least that’s what the crusaders tell me – I haven’t actually seen much evidence of that being the case).
I think it would be great if you could publish exploratory results, clearly labelled as such, and nobody would bat an eyelid. This should be accompanied by confirmation of those findings by other people. The whole thing should be a collaborative effort rather than the mudslinging contest it currently is.
Considering the fact that high profile journals like Nature now actually expect results to be replicated to be considered for publication is clearly a sign that things are changing although that’s not quite what I’m envisioning.
I thought scientists were “curious” and want to “try new things”, so why won’t they try the above scenario. They can then even investigate whether this leads to more robust results and stuff like that.
That’s what I am asking for! At the moment everything you say is hypothesis and conjecture. I want to see evidence before I am inclined to believe you.
Why is it that scientists are only “curious” and want to “try new things” when it comes to performing underpowered, random noise studies following the latest fad, and not want to explore important things like how to better science altogether !?!?
Who says that they are? My feeling from having all these debates is that scientists are pretty damn interested in doing better science…
Is it? Did we come closer to the truth?
Yes. I am not sure if preregistration and the wider discussion will actually make science better or worse. But ideas are being challenged and tried out and if they fail we will know they are mistakes. That is self-correction in action.
I don’t actually have a problem with people proposing and trying out new ideas. I simply oppose the new jerk reactions and mental breakdowns over reproducibility. Science works just as it should.
LikeLike
Just a quick addendum to our discussion. I just read this article about the whole STAP cell debacle: http://www.theguardian.com/science/2015/feb/18/haruko-obokata-stap-cells-controversy-scientists-lie
I think it is a very nice summary of the issues faced by science. It also clearly underlines that your argument that ‘psychology will never amount to anything useful’ is nonsense. Science fraud and issues in reproducibility are universal. Some people (like the BICEP2 team) deal with it better than others by making replication a collaborative effort that benefits the community as a whole. Anyway, I particularly like this quote as this mirrors very well what I’ve been saying:
“By contrast, the speed of Obokata’s undoing should make us feel more confident about the ability of science to correct itself.”
LikeLike