A defining feature of scientific knowledge is that it is replicable: another scientist should be able to repeat the procedure and re-generate the same results. Yet in reality, there are many obstacles to replication. The first obstacle is often: published procedures are not described sufficiently, to allow another scientist to reproduce them. Other obstacles to replication will be discussed in later classes (e.g. experimenter degrees of freedom and publication bias mean that published results offer an inflated estimate of effect sizes; professional incentives for scientists often don’t reward reproducing or being reproducible). Today we focus on the challenge of how to make scientific procedures reproducible.
Everyone should read the resources under ’everyone.’ After that please read the additional readings most relevant to you.
- Nosek, B. A., & Errington, T. M. (2020). “The best time to argue about what a replication means? Before you do it.” Nature 583, 518-520.
- Teytelman, L. (2018). “No more excuses for non-reproducible methods.” Nature 560, 411. Alternatively, watch a recent (2021) talk (the beginning is funny/sad).
Errington, T. M. (2019). “Reproducibility Project: Cancer Biology, Barriers to Replicability in the Process of Research.” Talk at the MetaScience Conference. Read the slides or watch the talk.
Banga, K., Benson, J., Bonacchi, N., et al. (2022). “Reproducibility of in-vivo electrophysiological measurements in mice.” bioRxiv.
- Barton, CM., et al.(2022). “How to make models more useful.” PNAS 119(35) e2202112119.
- Pineau, J. (2020). “The Machine Learning Reproducibility Checklist.”
- Gilmore, R. O., & Adolph, K. E. (2017). “Video can make behavioural science more reproducible.” Nature human behaviour, 1(7), 1-2.
- Frank, M. C., Bergelson, E., Bergmann, C., et al. (2017). “A collaborative approach to infant research: Promoting reproducibility, best practices, and theory‐building.” Infancy, 22(4), 421-435. Alternatively, watch this talk by Mike Frank.
Moody, J. W., Keister, L. A., & Ramos, M. C. (2022). “Reproducibility in the Social Sciences.” Annual Review of Sociology, 48.