New Nature paper tests the replicability of 274 claims from the social and behavioural sciences

We are proud to share that lab members Lusine Grigoryan and Isabelle Weissflog were part of a large collaborative study which tested the replicability of 274 claims across 54 journals in the social and behavioural sciences. The ‘replication crisis’ in psychology dawned in 2015 after the Reproducibility Project found only 36% of replications produced statistically significant effects, compared to 97% of the original results (Open Science Collaboration, 2015). Poor replicability has been associated with lower public trust in psychology (Wingen et al., 2020), and limits the generalisability of findings, so establishing its prevalence in the field and the causes of replication failures are critical for the future of social sciences research culture and science communication. The present study found that about half of the original findings replicated (successful replication rates ranged from 28.6% to 74.8% depending on the measure of replicability used). The full article can be read here and related articles from the SCORE project on reproducibility, robustness, and credibility can be found here.

Abstract

Pursuing replicability — independent evidence for previous claims — is important for creating generalizable knowledge. Here we attempted replications of 274 claims of positive results from 164 quantitative papers published from 2009 to 2018 in 54 journals in the social and behavioural sciences. Replications were high powered on average to detect the original effect size (median of 99.6%), used original materials when relevant and available, and were peer reviewed in advance through a standardized internal protocol. Replications showed statistically significant results in the original pattern for 151 of 274 claims (55.1% (95% confidence interval (CI) 49.2–60.9%)) and for 80.8 of 164 papers (49.3% (95% CI 43.8–54.7%)), weighed for replicating multiple claims per paper. We observed modest variation in replication rates across disciplines (42.5–63.1%), although some estimates had high uncertainty. The median Pearson’s r effect size was 0.25 (95% CI 0.21–0.27) for original studies and 0.10 (95% CI 0.09–0.13) for replication studies, an 82.4% (95% CI 67.8–88.2%) reduction in shared variance. Thirteen methods for evaluating replication success provided estimates ranging from 28.6% to 74.8% (median of 49.3%). Some decline in effect size and significance is expected based on power to detect original effects and regression to the mean because we replicated only positive results. We observe that challenges for replicability extend across social–behavioural sciences, illustrating the importance of identifying conditions that promote or inhibit replicability.

References:

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.

Wingen, T., Berkessel, J. B., & Englich, B. (2020). No Replication, No Trust? How Low Replicability Influences Trust in Psychology. Social Psychological and Personality Science, 11(4), 454-463. https://doi.org/10.1177/1948550619877412