In 2005, Dr. John Ioannidis, a well-known meta-researcher, published an article in PLoS Medicine called Why Most Published Research Findings Are False. This article caused a splash and has been making waves in the medical research community ever since. His paper is a bit technical, go for it if you can, but I recommend everyone at least read the less technical, narrative write-up about his research here in The Atlantic. He raises a number of very serious issues that have plagued the medical research community. I’ll try to summarize some of his concerns.
Publication bias, which means that if researchers find that an intervention had little or no effect, then those studies are less likely to get published. In other words, not publishing negative findings.
Overconfidence of studies that rely on statistical p-values of 0.05. This could lead to false positives as well as false negatives.
Lack of replication since many important studies are not repeated, or even ‘repeatable’, by other independent researchers to verify the results. Some major studies that have been replicated have found surprisingly different and even contradictory results.
Small study size and underpowered methodologies. The lower the number of subjects in the study, the less likely the results are true.
Selective outcome reporting, manipulation of data/fraud and financial conflicts of interest. The greater the financial interest, the less likely the study will be true.
The greater the flexibility in designing studies and in definitions, the less likely the research findings are to be true.
The reward systems within medical research, particularly in academia, incentivizes quantity of publications over quality of research.
These factors collectively have led Dr. Ioannidis to conclude that a large part of the evidence that doctors and healthcare providers have come to rely on, including major foundational studies used to treat patients, are frequently misleading, exaggerated, and often flat-out wrong.