Heterogeneity in effect size estimates
- Author(s)
- Felix Holzmeister, Magnus Johannesson, Robert Böhm, Anna Dreber, Jürgen Huber, Michael Kirchler
- Abstract
A typical empirical study involves choosing a sample, a research design, and an analysis path. Variation in such choices across studies leads to heterogeneity in results that introduce an additional layer of uncertainty, limiting the generalizability of published scientific findings. We provide a framework for studying heterogeneity in the social sciences and divide heterogeneity into population, design, and analytical heterogeneity. Our framework suggests that after accounting for heterogeneity, the probability that the tested hypothesis is true for the average population, design, and analysis path can be much lower than implied by nominal error rates of statistically significant individual studies. We estimate each type's heterogeneity from 70 multilab replication studies, 11 prospective meta-analyses of studies employing different experimental designs, and 5 multianalyst studies. In our data, population heterogeneity tends to be relatively small, whereas design and analytical heterogeneity are large. Our results should, however, be interpreted cautiously due to the limited number of studies and the large uncertainty in the heterogeneity estimates. We discuss several ways to parse and account for heterogeneity in the context of different methodologies.
- Organisation(s)
- Department of Occupational, Economic and Social Psychology, Department of Accounting, Innovation and Strategy
- External organisation(s)
- Leopold-Franzens-Universität Innsbruck, Stockholm School of Economics, University of Copenhagen
- Journal
- Proceedings of the National Academy of Sciences of the United States of America
- Volume
- 121
- No. of pages
- 10
- ISSN
- 0027-8424
- DOI
- https://doi.org/10.1073/pnas.2403490121
- Publication date
- 07-2024
- Peer reviewed
- Yes
- Austrian Fields of Science 2012
- 501021 Social psychology
- Keywords
- ASJC Scopus subject areas
- General
- Portal url
- https://ucrisportal.univie.ac.at/en/publications/heterogeneity-in-effect-size-estimates(2a4e2fa8-5998-4152-aa7c-8d5c3ad3cb8c).html