Barbara Osimani (LMU Munich): “Exact replication or varied evidence? Reliability, robustness and the reproducibility problem”

Abstract: The “Reproducibility Project: Psychology” by the Open Science Collaboration caused some stir among psychologists, methodologists as well as scientists, since less than half of the replicated studies succeeded in reproducing the results of the original ones. The APA has attributed this result to hidden moderators that rendered the replications ineffective. Also publication bias and low power have been identified as possible sources for such mismatch. While some analysts have provided formal confirmation for the plausibility of such explanations (Etz and Vandekerkhove, 2016), others have further insisted on the problem of noisy data and suggested that “to resolve the replication crisis in science we may need to consider each individual study in the context of an implicit meta-analysis” (Andrew Gelman).

I investigate these positions through the lenses of Bayesian epistemology, and in particular of recent results on the Variety of Evidence Thesis. This approach will turn out to be fruitful in investigating the interaction of reliability, independence of evidence, and replication in scientific inference and, more broadly, will cast a new light on the debate between advocates of a pluralist methodology in medical research, who insist on supporting hypotheses through various sources of evidence, and the contending view, represented by the Evidence Based Medicine paradigm, which relies on an “elitist” approach, where “best evidence” is searched for and exact replication is highly welcome.