Replication and Publication Bias

Why some published findings fail to hold up when repeated, how the bias toward publishing striking results distorts evidence, and what reforms are helping.

Science relies on findings that hold up when others check them. Replication means repeating a study, with the same data or new data, to see if the result holds. In the 2010s, a series of large replication projects in psychology, economics and medicine found that a worrying share of published results did not replicate as strongly as first reported.

Publication bias

One cause is publication bias. Journals and researchers tend to favour striking, positive results, such as “this programme works”, over null results showing no effect. If twenty teams test the same idea and one finds a positive result by chance, that one may be published while the nineteen null results sit in drawers. Readers then see only the lucky result.

P-hacking

A related problem is sometimes called p-hacking. Researchers make many choices when analysing data: which variables to include, which observations to drop, which outcomes to report. If they try many versions and report only the one that looks most impressive, they can produce results that appear strong but are really due to chance. This can happen without any intent to deceive.

The lucky coin

If you flip a coin ten times, getting eight or more heads happens only about 5 percent of the time. But if a hundred people each flip ten coins, several of them will get eight heads just by luck. If only those people report their results, it will look as though the coins are biased. Publication bias works in a similar way across many studies.

Reforms

Economics has responded with several reforms:

Pre-registration: researchers publicly record their plan for a study, including which outcomes they will measure, before seeing the data. This is now common for randomised trials, many of which are registered in the American Economic Association’s registry.
Data sharing: leading journals require authors to share data and code so others can check the results.
Replication studies: more journals publish attempts to replicate earlier findings.

A large 2016 project that re-ran 18 laboratory experiments published in two top economics journals found that 11 of the 18 replicated, with effects often smaller than originally reported.

Concluding that research is worthless

Replication problems do not mean research cannot be trusted. They mean that single studies should be read with caution, and that confidence should grow when several independent studies point the same way. The reforms under way are making evidence more reliable.

Key takeaways

Replication checks whether published findings hold up when repeated.
Publication bias favours striking results and hides null findings.
P-hacking can produce impressive-looking results that are really due to chance.
Pre-registration, data sharing and replication studies are improving reliability.