The Decline Effect, which seems to have appeared in almost every area of science, is a phenomenon scientists have noticed only recently in which the results which seemed initially to be very strong get weaker and weaker as studies replicate the original research.
Since nobody seems to be gaining from it, undue carelessness in carrying out the original research or even deliberate fraud do not seem to explain why the decline effect is happening. . The effect was initially even identified by someone trying to replicate his own research.
There might still, however, be a weakness in the way the scientific method is being implemented by the scientific community. The problem might be that we have inadvertently made it less and less worthwhile to publish research that does not show that whatever we are studying has some positive relationship to some apparent effect or result. For example, suppose an article studying Vitamin B12 supplements concludes that it can make a difference in reducing memory loss in old age. That’s good news. We can do something about some of that distressing memory loss. But if, on the other hand, the research finds no discernible differences, there is always the possibility that different research might still turn up something. Different doses, different combinations of vitamins, different populations, different measurements of cognitive functioning – all of them might turn up some result. In the meantime, the message is that there’s no new advice for dealing with our forgetfulness. Not nearly as many people are going to be interested in this headline information.
To put it in technical terms, it is difficult to get research published in professional journals that does not demonstrate what are called “statistically significant” results.
The essence of statistical analysis is deciding whether some event probably happened by chance or not. Although analyses have become extraordinarily sophisticated with the increasing computer power in the modern world, the principle remains the same.
For instance, walking down the street, there is one chance in 365 that the next person I see will have been born on the same day of the year as I was. There is one chance in two that if I flip a penny, it will come up heads. Science uses statistical analysis to decide if the effect connected to some variable happened by chance or is what is called “statistically significant.”
Did this group of people, for instance, who took an aspirin every day for the last ten years have lower levels of cancer than a similar group of people who didn’t regularly take aspirin? And if the first group did have lower levels of cancer, was that merely a chance difference or were the differences so great that the probability that it was chance are miniscule?
What the decline effect is suggesting is that results that at first look as if could almost certainly not have happened by chance gradually seem to look more and more like chance with repeated replication.
The problem is that if science is tending more and more to publish only those results that are statistically significant, then research which suggests that some variable does not have any effect does not tend to get known by the scientific community at large. If research showing that eating less red meat is associated with lower levels of cancer gets published, but if research showing that red meat consumption does not seem to be related to incidence of cancer does not, then there is going to be a bias in the publicly accessible research. So the fact that in reality the association between red meat and cancer might be either very small or not exist at all is going to take much longer to become evident.
The almost universally accepted level of probability that results are not caused by chance is 5%. In other words, to be statistically significant, there has to be less than 5 chances out of a 100 that it was a fluke outcome caused by chance. But that means that at least 5% of all research reporting a positive result is probably a result of chance and not a real effect at all.
Traditionally, the way to catch these errors has been through replication of research studies by a variety of different researchers. But if studies that are not statistically significant are difficult to get published, then these chance errors are not going to be found in the natural course of study.
Imagine you are a young, ambitious scientist eager to make a mark. Are you going to deliberately put your professional time and energy into doing research that is going to refute research which has already been published and is quite possibly lauded by the professional community? Is it the wisest thing to begin by pitting your findings against received wisdom? Wouldn’t it be better to strike out and find out something new and positive and statistically significant?
My strong suspicion is that the Decline Effect is the result of a widespread but indeliberate failure to adequately replicate initial research that reports some supposedly significant finding. And so it is taking us longer to sort out the wheat from the chaff, to identify those findings that are robust, and those that were based merely on chance.
The one thing science cannot afford to do is to let go of real, robust replication – including studies that report that they have been unable to find anything significant at all.
