Skip to main content

Research Repository

Advanced Search

Many Labs 2 : Investigating Variation in Replicability Across Sample and Setting

Klein, RA; et al

Many Labs 2 : Investigating Variation in Replicability Across Sample and Setting Thumbnail


Authors

RA Klein

et al



Contributors

Abstract

We conducted preregistered replications of 28 classic and contemporary published findings with protocols that were peer reviewed in advance to examine variation in effect magnitudes across sample and setting. Each protocol was administered to approximately half of 125 samples and 15,305 total participants from 36 countries and territories. Using conventional statistical significance (p < .05), fifteen (54%) of the replications provided evidence in the same direction and statistically significant as the original finding. With a strict significance criterion (p < .0001), fourteen (50%) provide such evidence reflecting the extremely high powered design. Seven (25%) of the replications had effect sizes larger than the original finding and 21 (75%) had effect sizes smaller than the original finding. The median comparable Cohen’s d effect sizes for original findings was 0.60 and for replications was 0.15. Sixteen replications (57%) had small effect sizes (< .20) and 9 (32%) were in the opposite direction from the original finding. Across settings, 11 (39%) showed significant heterogeneity using the Q statistic and most of those were among the findings eliciting the largest overall effect sizes; only one effect that was near zero in the aggregate showed significant heterogeneity. Only one effect showed a Tau > 0.20 indicating moderate heterogeneity. Nine others had a Tau near or slightly above 0.10 indicating slight heterogeneity. In moderation tests, very little heterogeneity was attributable to task order, administration in lab versus online, and exploratory WEIRD versus less WEIRD culture comparisons. Cumulatively, variability in observed effect sizes was more attributable to the effect being studied than the sample or setting in which it was studied.

Data, materials and code available at: <a href="https://osf.io/8cd4r/"> https://osf.io/8cd4r/</a>

Citation

Klein, R., & et al. (2018). Many Labs 2 : Investigating Variation in Replicability Across Sample and Setting. https://doi.org/10.31234/osf.io/9654g

Journal Article Type Article
Online Publication Date Nov 19, 2018
Publication Date Nov 19, 2018
Deposit Date Nov 21, 2018
Publicly Available Date Nov 21, 2018
Journal PsyArXiv Preprints
DOI https://doi.org/10.31234/osf.io/9654g
Publisher URL https://doi.org/10.31234/osf.io/9654g
Related Public URLs https://psyarxiv.com/9654g/
https://www.psychologicalscience.org/publications/many-labs-2

Files

ML2 (1).pdf (4.6 Mb)
PDF

Version
Published under Creative Commons CC0 1.0 Universal licence





You might also like



Downloadable Citations