The Power of Bias in Economics Research

Ioannidis, John P. A.; Stanley, T. D.; Doucouliagos, Hristos

doi:10.1111/ecoj.12461

Cited by 476 publications

(507 citation statements)

References 79 publications

(153 reference statements)

Supporting

Mentioning

479

Contrasting

Unclassified

Order By: Relevance

“…First, the median power in intelligence research seems higher than the median power estimated in neuroscience (8-31%; Button et al, 2013), psychology (between 12% and 44%; Szucs & Ioannidis, 2017;Stanley et al, 2017) , behavioral ecology and animal research (13-16% for a small effect and 40-47% for a medium effect; Jennions & Moller, 2003), and economics (18%; Ioannidis, Stanley, & Doucouliagos, 2017), but slightly lower than socialpersonality research (50% for r = .20; Fraley & Vazire, 2014). Second, we did not find clear trends in effect sizes over time, which might indicate that the field of intelligence research is less susceptible to time-lag biases such as the decline effect or the early-extreme effect .…”

Section: Discussionmentioning

confidence: 91%

Research on Research: A Meta-Scientific Study of Problems and Solutions in Psychological Science

Nuijten¹

2018

Preprint

View full text Add to dashboard Cite

SUMMARY DOCTORAL DISSERTATION: Psychology is facing a “replication crisis”. Many psychological findings could not be replicated in novel samples, which lead to the growing concern that many published findings are overly optimistic or even false. In this dissertation, we investigated potential indicators of problems in the published psychological literature. In Part I of this dissertation, we looked at inconsistencies in reported statistical results in published psychology papers. To facilitate our research, we developed the free tool statcheck; a “spellchecker” for statistics. In Part II, we investigated bias in published effect sizes. We showed that in the presence of publication bias, the overestimation of effects can become worse if you combine studies. Indeed, in meta-analyses from the social sciences we found strong evidence that published effects are overestimated. These are worrying findings, and it is important to think about concrete solutions to improve the quality of psychological research. Some of the solutions we propose are preregistration, replication, and transparency. We argue that to select the best strategies to improve psychological science, we need research on research: meta-research.

show abstract

Section: Discussionmentioning

confidence: 91%

Research on Research: A Meta-Scientific Study of Problems and Solutions in Psychological Science

Nuijten¹

2018

Preprint

View full text Add to dashboard Cite

show abstract

“…However, the evidence base reported in leading journals is far from ideal. Indeed, leading journals are subject to publication selection bias (leaving some of the evidence base unreported because the estimates are not statistically significant or because they do not conform to researchers' expectations), and there is a worrying inability to replicate key results [2]. The situation in other journals is no better.…”

Section: Discussion Of Pros and Consmentioning

confidence: 99%

“…Empirical studies have adequate statistical power when they can detect a genuine empirical effect; this requires low rates of false negatives (predictions that are true but are predicted as false). A survey of 159 meta-regression analyses of some 6,700 empirical studies in economics found that such research is greatly underpowered: statistical power in economics is typically no more than 18% (compared to the ideal of 80%), with almost half of the 159 empirical economics areas surveyed having 90% or more of the reported results from underpowered studies [2]. Meta-regression analysis increases statistical power by combining the results from…”

Section: Increasing Statistical Powermentioning

confidence: 99%

“…For example, the survey of 159 economics topics found empirical economics to be highly contaminated by selection bias, with a majority of the reported effects exaggerated by a factor of at least two, conservatively estimated [2]. It goes without saying that such artificial inflation of elasticities has major repercussions for perceived policy effectiveness.…”

Section: Correcting or Reducing Publication Selection Biasmentioning

confidence: 99%

See 1 more Smart Citation

Meta-regression analysis: Producing credible estimates from diverse evidence

Doucouliagos¹

2016

izawol

View full text Add to dashboard Cite

show abstract

“…Several studies from across the social sciences have documented a variety of issues in existing research, including publication bias (Sterling (1959), Rosenthal (1979), Franco, Malhotra, and Simonovits (2014)); specification search (Simmons, Nelson, and Simonsohn (2011), Humphreys, de la Sierra, and van der Windt (2013), Brodeur et al (2016); bias and failures to replicate (Open Science Collaboration (2015), Chang and Li (2015), Camerer et al (2016), Ioannidis, Stanley, and Doucouliagos (2017)); and outright fraud (Simonsohn (2013) and Broockman, Kalla, and Aronow (2015)). 1 This research raises concerns about the validity of empirical work in economics and in social science more generally.…”

Section: Introductionmentioning

confidence: 99%

Improving transparency in observational social science research: A pre-analysis plan approach

Burlig

2018

Economics Letters

View full text Add to dashboard Cite

Social science research has undergone a credibility revolution, but these gains are at risk due to problematic research practices. Existing research on transparency has centered around randomized controlled trials, which constitute only a small fraction of research in economics. In this paper, I discuss three scenarios in which study preregistration can be credibly applied in non-experimental settings: cases where researchers collect their own data; prospective studies; and research using restricted-access data. Finally, I outline suggested contents for observational pre-analysis plans, and highlight where these plans should deviate from those designed for experimental research.

show abstract

The Power of Bias in Economics Research

Cited by 476 publications

References 79 publications

Research on Research: A Meta-Scientific Study of Problems and Solutions in Psychological Science

Research on Research: A Meta-Scientific Study of Problems and Solutions in Psychological Science

Meta-regression analysis: Producing credible estimates from diverse evidence

Improving transparency in observational social science research: A pre-analysis plan approach

Contact Info

Product

Resources

About