Feature Article
Full Access
The Power of Bias in Economics Research
We acknowledge support from the Czech Science Foundation (grant 15‐02411S). The Meta‐Research Innovation Center at Stanford (METRICS) is supported by a grant by the Laura and John Arnold Foundation and the work of John Ioannidis is also supported by an unrestricted gift from Sue and Bob O'Donnell. We are also grateful for the comments by the participants at the Meta‐Analysis of Economics Research Network (MAER‐Net) 2015 colloquium, Prague, CZ.
Abstract
We investigate two critical dimensions of the credibility of empirical economics research: statistical power and bias. We survey 159 empirical economics literatures that draw upon 64,076 estimates of economic parameters reported in more than 6,700 empirical studies. Half of the research areas have nearly 90% of their results under‐powered. The median statistical power is 18%, or less. A simple weighted average of those reported results that are adequately powered (power ≥ 80%) reveals that nearly 80% of the reported effects in these empirical economics literatures are exaggerated; typically, by a factor of two and with one‐third inflated by a factor of four or more.
Citing Literature
Number of times cited according to CrossRef: 104
- Sophie C. Schmidt, Ben Marwick, Tool-Driven Revolutions in Archaeological Science, Journal of Computer Applications in Archaeology, 10.5334/jcaa.29, 3, 1, (18-32), (2020).
- Zohid Askarov, Hristos Doucouliagos, A meta-analysis of the effects of remittances on household education expenditure, World Development, 10.1016/j.worlddev.2019.104860, 129, (104860), (2020).
- Alberto Abadie, Statistical Nonsignificance in Empirical Economics, American Economic Review: Insights, 10.1257/aeri.20190252, 2, 2, (193-208), (2020).
- Brian T. Yates, Research on Improving Outcomes and Reducing Costs of Psychological Interventions: Toward Delivering the Best to the Most for the Least, Annual Review of Clinical Psychology, 10.1146/annurev-clinpsy-071519-110415, 16, 1, (125-150), (2020).
- Christoph Kubitza, Vijesh V. Krishna, Urs Schulthess, Meha Jain, Estimating adoption and impacts of agricultural management practices in developing countries using satellite data. A scoping review, Agronomy for Sustainable Development, 10.1007/s13593-020-0610-2, 40, 3, (2020).
- Michael Kossmeier, Ulrich S. Tran, Martin Voracek, Power-Enhanced Funnel Plots for Meta-Analysis, Zeitschrift für Psychologie, 10.1027/2151-2604/a000392, 228, 1, (43-49), (2020).
- Andreas Vigren, Roger Pyddoke, The impact on bus ridership of passenger incentive contracts in public transport, Transportation Research Part A: Policy and Practice, 10.1016/j.tra.2020.03.003, 135, (144-159), (2020).
- , The Production of Knowledge, 10.1017/9781108762519, (2020).
- Tom E. Hardwicke, Stylianos Serghiou, Perrine Janiaud, Valentin Danchev, Sophia Crüwell, Steven N. Goodman, John P.A. Ioannidis, Calibrating the Scientific Ecosystem Through Meta-Research, Annual Review of Statistics and Its Application, 10.1146/annurev-statistics-031219-041104, 7, 1, (11-37), (2020).
- Zareh Asatryan, Annika Havlik, Friedrich Heinemann, Justus Nover, Biases in Fiscal Multiplier Estimates, European Journal of Political Economy, 10.1016/j.ejpoleco.2020.101861, (101861), (2020).
- References, A Fast and Frugal Finance, 10.1016/B978-0-12-812495-6.00019-7, (233-248), (2020).
- Fast-and-frugal asset pricing, A Fast and Frugal Finance, 10.1016/B978-0-12-812495-6.00014-8, (133-155), (2020).
- Mahesh Dahal, Nathan Fiala, What do we know about the impact of microfinance? The problems of statistical power and precision, World Development, 10.1016/j.worlddev.2019.104773, 128, (104773), (2020).
- Stephan B. Bruns, Martin Kalthaus, Flexibility in the selection of patent counts: Implications for p-hacking and evidence-based policymaking, Research Policy, 10.1016/j.respol.2019.103877, 49, 1, (103877), (2020).
- Long Chu, R.Quentin Grafton, Water pricing and the value-add of irrigation water in Vietnam: Insights from a crop choice model fitted to a national household survey, Agricultural Water Management, 10.1016/j.agwat.2019.105881, 228, (105881), (2020).
- Alina Cazachevici, Tomas Havranek, Roman Horvath, Remittances and economic growth: A meta-analysis, World Development, 10.1016/j.worlddev.2020.105021, 134, (105021), (2020).
- Mehmet Ugur, Sefa Awaworyi Churchill, Hoang M. Luong, What do we know about R&D; spillovers and productivity? Meta-analysis evidence on heterogeneity and statistical power, Research Policy, 10.1016/j.respol.2019.103866, 49, 1, (103866), (2020).
- Julian Reif, David Chan, Damon Jones, Laura Payne, David Molitor, Effects of a Workplace Wellness Program on Employee Health, Health Beliefs, and Medical Use, JAMA Internal Medicine, 10.1001/jamainternmed.2020.1321, (2020).
- Guillaume Coqueret, Stock-specific sentiment and return predictability, Quantitative Finance, 10.1080/14697688.2020.1736314, (1-21), (2020).
- Diana W. Thomas, Michael D. Thomas, Behavioral symmetry, rent seeking, and the Republic of Science, Public Choice, 10.1007/s11127-020-00807-4, (2020).
- EMILIE COURTIN, SOOYOUNG KIM, SHANSHAN SONG, WENYA YU, PETER MUENNIG, Can Social Policies Improve Health? A Systematic Review and Meta‐Analysis of 38 Randomized Trials, The Milbank Quarterly, 10.1111/1468-0009.12451, 0, 0, (2020).
- Tomáš Havránek, T. D. Stanley, Hristos Doucouliagos, Pedro Bom, Jerome Geyer‐Klingeberg, Ichiro Iwasaki, W. Robert Reed, Katja Rost, R. C. M. Aert, REPORTING GUIDELINES FOR META‐ANALYSIS IN ECONOMICS, Journal of Economic Surveys, 10.1111/joes.12363, 0, 0, (2020).
- Panagiotis Tsigaris, Jaime A. Teixeira da Silva, Reproducibility issues with correlating Beall-listed publications and research awards at a small Canadian business school, Scientometrics, 10.1007/s11192-020-03353-4, (2020).
- Mojmir Hampl, Tomas Havranek, Zuzana Irsova, Foreign capital and domestic productivity in the Czech Republic: a meta-regression analysis, Applied Economics, 10.1080/00036846.2020.1726864, (1-10), (2020).
- Guillaume Rochefort-Maranda, Inflated effect sizes and underpowered tests: how the severity measure of evidence is affected by the winner’s curse, Philosophical Studies, 10.1007/s11098-020-01424-z, (2020).
- Sarah Tahamont, Zubin Jelveh, Aaron Chalfin, Shi Yan, Benjamin Hansen, Dude, Where’s My Treatment Effect? Errors in Administrative Data Linking and the Destruction of Statistical Power in Randomized Experiments, Journal of Quantitative Criminology, 10.1007/s10940-020-09461-x, (2020).
- Shinichi Nakagawa, Malgorzata Lagisz, Rose E. O'Dea, Joanna Rutkowska, Yefeng Yang, Daniel W. A. Noble, Alistair M. Senior, The orchard plot: Cultivating a forest plot for use in ecology, evolution, and beyond, Research Synthesis Methods, 10.1002/jrsm.1424, 0, 0, (2020).
- Moriah Bostian, Cinzia Daraio, Shawna Grosskopf, Giancarlo Ruocco, William L. Weber, Sources and uses of knowledge in a dynamic network technology, International Transactions in Operational Research, 10.1111/itor.12741, 27, 4, (1821-1844), (2019).
- Hristos Doucouliagos, Patrice Laroche, Douglas L. Kruse, T. D. Stanley, Is Profit Sharing Productive? A Meta‐Regression Analysis, British Journal of Industrial Relations, 10.1111/bjir.12483, 58, 2, (364-395), (2019).
- Stephanie M. Rizio, Ahmed Skali, How often do dictators have positive economic effects? Global evidence, 1858–2010, The Leadership Quarterly, 10.1016/j.leaqua.2019.06.003, (2019).
- Marco Colagrossi, Domenico Rossignoli, Mario A. Maggioni, Does democracy cause growth? A meta-analysis (of 2000 regressions), European Journal of Political Economy, 10.1016/j.ejpoleco.2019.101824, (101824), (2019).
- Jerome Olsen, Johanna Mosen, Martin Voracek, Erich Kirchler, Research practices and statistical reporting quality in 250 economic psychology master's theses: a meta-research investigation, Royal Society Open Science, 10.1098/rsos.190738, 6, 12, (190738), (2019).
- Stephan B. Bruns, John P.A. Ioannidis, Determinants of economic growth: Different time different answer?, Journal of Macroeconomics, 10.1016/j.jmacro.2019.103185, (103185), (2019).
- Bernardo Mueller, Why public policies fail: Policymaking under complexity, EconomiA, 10.1016/j.econ.2019.11.002, (2019).
- S. V. Bruton, M. Brown, D. F. Sacco, R. Didlake, Testing an active intervention to deter researchers’ use of questionable research practices, Research Integrity and Peer Review, 10.1186/s41073-019-0085-3, 4, 1, (2019).
- Alessandro Roncaglia, , The Age of Fragmentation, 10.1017/9781108777766, (2019).
- Eva Vivalt, Specification Searching and Significance Inflation Across Time, Methods and Disciplines, Oxford Bulletin of Economics and Statistics, 10.1111/obes.12289, 81, 4, (797-816), (2019).
- Jérôme Hubler, Christine Louargant, Patrice Laroche, Jean‐Noёl Ory, HOW DO RATING AGENCIES’ DECISIONS IMPACT STOCK MARKETS? A META‐ANALYSIS, Journal of Economic Surveys, 10.1111/joes.12317, 33, 4, (1173-1198), (2019).
- Sanjay Poudel, Fadi Shehadeh, Ioannis M Zacharioudakis, Giannoula S Tansarli, Fainareti N Zervou, Markos Kalligeros, Robertus van Aalst, Ayman Chit, Eleftherios Mylonakis, The Effect of Influenza Vaccination on Mortality and Risk of Hospitalization in Patients With Heart Failure: A Systematic Review and Meta-analysis, Open Forum Infectious Diseases, 10.1093/ofid/ofz159, 6, 4, (2019).
- Eszter Czibor, David Jimenez‐Gomez, John A. List, The Dozen Things Experimental Economists Should Do (More of), Southern Economic Journal, 10.1002/soej.12392, 86, 2, (371-432), (2019).
- Pedro R. D. Bom, Heiko Rachinger, A kinked meta‐regression model for publication bias correction, Research Synthesis Methods, 10.1002/jrsm.1352, 10, 4, (497-514), (2019).
- Meng Li, Anirban Basu, Caroline S. Bennette, David L. Veenstra, Louis P. Garrison, Do cancer treatments have option value? Real‐world evidence from metastatic melanoma, Health Economics, 10.1002/hec.3899, 28, 7, (855-867), (2019).
- Daniel Grossman, Urnair Khalil, Arijit Ray, Terrorism and early childhood health outcomes: Evidence from Pakistan, Social Science & Medicine, 10.1016/j.socscimed.2019.112453, (112453), (2019).
- Isaiah Andrews, Maximilian Kasy, Identification of and Correction for Publication Bias, American Economic Review, 10.1257/aer.20180310, 109, 8, (2766-2794), (2019).
- Evan C. Carter, Felix D. Schönbrodt, Will M. Gervais, Joseph Hilgard, Correcting for Bias in Psychology: A Comparison of Meta-Analytic Methods, Advances in Methods and Practices in Psychological Science, 10.1177/2515245919847196, 2, 2, (115-144), (2019).
- Katharine O. Strunk, Bradley D. Marianno, Negotiating the Great Recession: How Teacher Collective Bargaining Outcomes Change in Times of Financial Duress, AERA Open, 10.1177/2332858419855089, 5, 2, (233285841985508), (2019).
- Tomas Havranek, Anna Sokolova, Do consumers really follow a rule of thumb? Three thousand estimates from 144 studies say “Probably not”, Review of Economic Dynamics, 10.1016/j.red.2019.05.004, (2019).
- Eszter Czibor, David Jimenez-Gomez, John A. List, The Dozen Things Experimental Economists Should Do (More Of), SSRN Electronic Journal, 10.2139/ssrn.3313734, (2019).
- Kenneth Button, applied Economics and Understanding Trends in Air Transportation Policy, Transport Policy, 10.1016/j.tranpol.2019.04.022, (2019).
- Shu Wing Chan, Steven Schilizzi, Md Sayed Iftekhar, Raymond Da Silva Rosa, Web-based experimental economics software: How do they compare to desirable features?, Journal of Behavioral and Experimental Finance, 10.1016/j.jbef.2019.04.007, (2019).
- Eirik Strømland, Preregistration and reproducibility, Journal of Economic Psychology, 10.1016/j.joep.2019.01.006, (2019).
- Mohsen Javdani, Ha-Joon Chang, Who Said or What Said? Estimating Ideological Bias in Views Among Economists, SSRN Electronic Journal, 10.2139/ssrn.3356309, (2019).
- Stephan B. Bruns, Igor Asanov, Rasmus Bode, Melanie Dunger, Christoph Funk, Sherif M. Hassan, Julia Hauschildt, Dominik Heinisch, Karol Kempa, Johannes König, Johannes Lips, Matthias Verbeck, Eva Wolfschütz, Guido Buenstorf, Reporting errors and biases in published empirical findings: Evidence from innovation research, Research Policy, 10.1016/j.respol.2019.05.005, 48, 9, (103796), (2019).
- Bernard S. Black, Alex Hollingsworth, Leticia Nunes, Kosali Ilayperuma Simon, The Effect of Health Insurance on Mortality: What Can We Learn from the Affordable Care Act Coverage Expansions?, SSRN Electronic Journal, 10.2139/ssrn.3368187, (2019).
- William H. B. McAuliffe, Evan C. Carter, Juliana Berhane, Alexander C. Snihur, Michael E. McCullough, Is Empathy the Default Response to Suffering? A Meta-Analytic Evaluation of Perspective Taking’s Effect on Empathic Concern, Personality and Social Psychology Review, 10.1177/1088868319887599, (108886831988759), (2019).
- Jae H. Kim, In Choi, Choosing the Level of Significance: A Decision‐theoretic Approach, Abacus, 10.1111/abac.12172, 0, 0, (2019).
- Peter Kedron, Amy E. Frazier, Andrew B. Trgovac, Trisalyn Nelson, A. Stewart Fotheringham, Reproducibility and Replicability in Geographical Analysis, Geographical Analysis, 10.1111/gean.12221, 0, 0, (2019).
- Mojmir Hampl, Tomas Havranek, Central Bank Equity as an Instrument of Monetary Policy, Comparative Economic Studies, 10.1057/s41294-019-00092-1, (2019).
- Jaime A. Teixeira da Silva, Panagiotis Tsigaris, How hyped media and misleading editorials can influence impressions about Beall’s listsof “predatory” publications, Journal of Information, Communication and Ethics in Society, 10.1108/JICES-06-2018-0059, (2019).
- Thi Mai Lan Nguyen, Output Effects of Monetary Policy in Emerging and Developing Countries: Evidence from a Meta-Analysis, Emerging Markets Finance and Trade, 10.1080/1540496X.2019.1601081, (1-18), (2019).
- Andreas Heinrich, Felix Herrmann, Heiko Pleines, Transparency and quality assessment of research data in post-Soviet area studies: The potential of an interactive online platform, Journal of Eurasian Studies, 10.1177/1879366519850698, (187936651985069), (2019).
- Sophie Thoyer, Raphaële Préget, Enriching the CAP evaluation toolbox with experimental approaches: introduction to the special issue, European Review of Agricultural Economics, 10.1093/erae/jbz024, (2019).
- Leah H. Palm-Forster, Paul J. Ferraro, Nicholas Janusch, Christian A. Vossler, Kent D. Messer, Behavioral and Experimental Agri-Environmental Research: Methodological Challenges, Literature Gaps, and Recommendations, Environmental and Resource Economics, 10.1007/s10640-019-00342-x, (2019).
- Michael Nippa, Jeffrey J Reuer, On the future of international joint venture research, Journal of International Business Studies, 10.1057/s41267-019-00212-0, (2019).
- Filiz Yesilyurt, M Ensar Yesilyurt, Meta-analysis, military expenditures and growth, Journal of Peace Research, 10.1177/0022343318808841, (002234331880884), (2019).
- Jasney Cogua, Kai Yin Ho, W. Alex Mason, The Peril and Promise of Racial and Ethnic Subgroup Analysis in Health Disparities Research, Journal of Evidence-Based Social Work, 10.1080/26408066.2019.1591317, (1-11), (2019).
- Ryan H. Murphy, The rationality of literal Tide Pod consumption, Journal of Bioeconomics, 10.1007/s10818-019-09285-1, (2019).
- Pierre Mandon, Antoine Cazals, POLITICAL BUDGET CYCLES: MANIPULATION BY LEADERS VERSUS MANIPULATION BY RESEARCHERS? EVIDENCE FROM A META‐REGRESSION ANALYSIS, Journal of Economic Surveys, 10.1111/joes.12263, 33, 1, (274-308), (2018).
- Michèle B Nuijten, Practical tools and strategies for researchers to increase replicability, Developmental Medicine & Child Neurology, 10.1111/dmcn.14054, 61, 5, (535-539), (2018).
- Kenneth Button, The value and challenges of using meta-analysis in transportation economics, Transport Reviews, 10.1080/01441647.2018.1464078, 39, 3, (293-308), (2018).
- Stephan B. Bruns, David I. Stern, Lag length selection and p-hacking in Granger causality testing: prevalence and performance of meta-regression models, Empirical Economics, 10.1007/s00181-018-1446-3, 56, 3, (797-830), (2018).
- Nauro F. Campos, Jarko Fidrmuc, Iikka Korhonen, Business cycle synchronisation and currency unions: A review of the econometric evidence using meta-analysis, International Review of Financial Analysis, 10.1016/j.irfa.2018.11.012, (2018).
- Felix Gottschalk, Markets for Advice. An Experimental Investigation, SSRN Electronic Journal, 10.2139/ssrn.3113528, (2018).
- Tomas Havranek, Zuzana Irsova, Olesia Zeynalova, Tuition Fees and University Enrolment: A Meta‐Regression Analysis, Oxford Bulletin of Economics and Statistics, 10.1111/obes.12240, 80, 6, (1145-1184), (2018).
- W. Robert Reed, A Primer on the ‘Reproducibility Crisis’ and Ways to Fix It, Australian Economic Review, 10.1111/1467-8462.12262, 51, 2, (286-300), (2018).
- John P.A. Ioannidis, Randomized controlled trials: Often flawed, mostly useless, clearly indispensable: A commentary on Deaton and Cartwright, Social Science & Medicine, 10.1016/j.socscimed.2018.04.029, 210, (53-56), (2018).
- Samuel Johnson, Toward a Cognitive Science of Markets: Economic Agents As Sense-Makers, SSRN Electronic Journal, 10.2139/ssrn.3250031, (2018).
- W. Alex Mason, Jasney Cogua-Lopez, Charles B. Fleming, Lawrence M. Scheier, Challenges Facing Evidence-Based Prevention: Incorporating an Abductive Theory of Method, Evaluation & the Health Professions, 10.1177/0163278718772879, 41, 2, (155-182), (2018).
- Jesper W. Schneider, NHST is still logically flawed, Scientometrics, 10.1007/s11192-018-2655-4, 115, 1, (627-635), (2018).
- Stephanos Papadamou, Νikolaos A. Kyriazis, Panayiotis G. Tzeremes, Unconventional monetary policy effects on output and inflation: A meta-analysis, International Review of Financial Analysis, 10.1016/j.irfa.2018.11.015, (2018).
- Santiago Schnell, “Reproducible” Research in Mathematical Sciences Requires Changes in our Peer Review Culture and Modernization of our Current Publication Approach, Bulletin of Mathematical Biology, 10.1007/s11538-018-0500-9, 80, 12, (3095-3105), (2018).
- Saltelli Andrea, Why science’s crisis should not become a political battling ground, Futures, 10.1016/j.futures.2018.07.006, 104, (85-90), (2018).
- Jaime A. Teixeira da Silva, Panagiotis Tsigaris, What Value Do Journal Whitelists and Blacklists Have in Academia?, The Journal of Academic Librarianship, 10.1016/j.acalib.2018.09.017, (2018).
- Hristos Doucouliagos, Martin Paldam, T.D. Stanley, Skating on thin evidence: Implications for public policy, European Journal of Political Economy, 10.1016/j.ejpoleco.2018.03.004, 54, (16-25), (2018).
- Martin Paldam, A model of the representative economist, as researcher and policy advisor, European Journal of Political Economy, 10.1016/j.ejpoleco.2018.03.005, 54, (5-15), (2018).
- Guillaume Coqueret, The Economic Value of Firm-Specific News Sentiment, SSRN Electronic Journal, 10.2139/ssrn.3248925, (2018).
- Christopher M. Snyder, Ran Zhuo, Sniff Tests in Economics: Aggregate Distribution of Their Probability Values and Implications for Publication Bias, SSRN Electronic Journal, 10.2139/ssrn.3247630, (2018).
- Fiona Burlig, Improving transparency in observational social science research: A pre-analysis plan approach, Economics Letters, 10.1016/j.econlet.2018.03.036, 168, (56-60), (2018).
- Timothy Powell-Jackson, Calum Davey, Edoardo Masset, Shari Krishnaratne, Richard Hayes, Kara Hanson, James R Hargreaves, Trials and tribulations: cross-learning from the practices of epidemiologists and economists in the evaluation of public health interventions, Health Policy and Planning, 10.1093/heapol/czy028, 33, 5, (702-706), (2018).
- Kewei Hou, Chen Xue, Lu Zhang, Replicating Anomalies, The Review of Financial Studies, 10.1093/rfs/hhy131, (2018).
- George A. Akerlof, Pascal Michaillat, Persistence of false paradigms in low-power sciences, Proceedings of the National Academy of Sciences, 10.1073/pnas.1816454115, (201816454), (2018).
- Dimitrios Bakas, Georgios Chortareas, Georgios Magkonis, Volatility and growth: a not so straightforward relationship, Oxford Economic Papers, 10.1093/oep/gpy065, (2018).
- Hristos Doucouliagos, Richard B. Freeman, Patrice Laroche, T. D. Stanley, How Credible Is Trade Union Research? Forty Years of Evidence on the Monopoly–Voice Trade-Off, ILR Review, 10.1177/0019793917751144, 71, 2, (287-305), (2017).
- Zacharias Maniadis, Fabio Tufano, John A. List, To Replicate or Not To Replicate? Exploring Reproducibility in Economics through the Lens of a Model and a Pilot Study, The Economic Journal, 10.1111/ecoj.12527, 127, 605, (F209-F235), (2017).
- Zacharias Maniadis, Fabio Tufano, The Research Reproducibility Crisis and Economics of Science, The Economic Journal, 10.1111/ecoj.12526, 127, 605, (F200-F208), (2017).
- Alfredo Di Tillio, Marco Ottaviani, Peter Norman Sørensen, Persuasion Bias in Science: Can Economics Help?, The Economic Journal, 10.1111/ecoj.12515, 127, 605, (F266-F304), (2017).
- Campbell R. Harvey, Yan Liu, False (and Missed) Discoveries in Financial Economics, SSRN Electronic Journal, 10.2139/ssrn.3073799, (2017).
- Craig A. Gallet, Hristos Doucouliagos, The impact of healthcare spending on health outcomes: A meta-regression analysis, Social Science & Medicine, 10.1016/j.socscimed.2017.02.024, 179, (9-17), (2017).
- David McMillan, Replication studies, Cogent Economics & Finance, 10.1080/23322039.2017.1410940, 5, 1, (2017).
- Nicolas Vallois, Dorian Jullien, Estimating Rationality in Economics: A History of Statistical Methods in Experimental Economics, SSRN Electronic Journal, 10.2139/ssrn.2990987, (2017).
