The preregistration revolution

Deception researchers widely acknowledge that cues to deception -observable behaviors that may differ between truthful and deceptive messagestend to be weak. Nevertheless, several deception cues have been reported with unusually large effect sizes, and some researchers have advocated the use of such cues as tools for detecting deceit and assessing credibility in practical contexts. Examining data from empirical deception cue research and using a series of Monte Carlo simulations, I demonstrate that many estimated effect sizes of deception cues may be greatly inflated by publication bias, small numbers of estimates, and low power. Indeed, simulations indicate the informational value of the present deception literature is quite low, such that it is not possible to determine whether any given effect is real or a false positive. I warn against the hazards of relying on potentially illusory cues to deception and offer some recommendations for improving the state of the science of deception. A preprint of this document is available at https://osf.io/xt8fq/."That Marionette," continued the Talking Cricket, "is a rascal of the worst kind." (Collodi, 1883) In an effort to temper our hopes of catching lies with unerring accuracy, deception researchers often say there is no "Pinocchio's nose" (e.g., Frank, Menasco, & O'Sullivan, 2008;Hartwig & Bond, 2011;Vrij, 2006Vrij, , 2004. That is, there is no behavior that perfectly discriminates between truthful and deceptive messages. For all the talk about Pinocchio's nose, there are other more urgent lessons from the story of Pinocchio pertinent to deception research.In his adventures, Pinocchio causes mischief and mayhem, and among his many foibles is the ease with which he is tempted to do what is easy and immediately satisfying but ultimately dangerous, rather than what is difficult and responsible. * timothy.luke@psy.gu.se. For their helpful comments, suggestions, and inspiration, I am grateful to Fabi Alceste, Karl Ask, Charlie Bond, Will Crozier, Bella DePaulo, Emelie Ernberg, Pär Anders Granhag, Maria Hartwig, Lorraine Hope, Emily Joseph, Louise Jupe, Erik Mac Giolla, Patty Sanchez, Rebecca Willén, and three anonymous reviewers, as well as the many supportive others who provided encouragement and tolerated my ravings and rantings.

In this article, we argue that the debate on the poor reproducibility of scientific research has overlooked an entire field: replication is also possible and desirable in the humanities. So far, the debate on replicability has been carried out primarily in the biomedical, natural and social sciences. It turns out that, for a wide variety of reasons, many of which lead to selective reporting, a large proportion of studies in these fields are not replicable, sometimes as many as 70 percent. In this paper, we leave these fields mostly aside, since they have been extensively addressed in the recent literature, and turn to the humanities. First, we distinguish between replicability and replication. Subsequently, we defend the view that replication is entirely possible in the humanities: it meets all the criteria that have been identified for biomedical, natural and social science research. The uniqueness of many research objects in the humanities does not present an obstacle to this. We also explain why replication is desirable and urgently needed in the humanities. Finally, we give various practical guidelines for how replication in the humanities could be carried out, such as focusing on the replication of cornerstone studies or a random selection of published research in a sub-discipline, and opting, if possible, for a conceptual replication, so that triangulation becomes possible.

Gelman and Loken (2013, 2014) proposed that when researchers base their statistical analyses on the idiosyncratic characteristics of a specific sample (e.g., a nonlinear transformation of a variable because it is skewed), they open up alternative analysis paths in potential replications of their study that are based on different samples (i.e., no transformation of the variable because it is not skewed). These alternative analysis paths count as additional (multiple) tests and, consequently, they increase the probability of making a Type I error during hypothesis testing. The present article considers this forking paths problem and evaluates four potential solutions that might be used in psychology and other fields: (a) adjusting the prespecified alpha level, (b) preregistration, (c) sensitivity analyses, and (d) abandoning the Neyman-Pearson approach. It is concluded that although preregistration and sensitivity analyses are effective solutions to p-hacking, they are ineffective against result-neutral forking paths, such as those caused by transforming data. Conversely, although adjusting the alpha level cannot address p-hacking, it can be effective for result-neutral forking paths. Finally, abandoning the Neyman-Pearson approach represents a further solution to the forking paths problem.Keywords: forking paths; null hypothesis significance testing; preregistration; replication crisis; sensitivity analyses Copyright © 2017, American Psychological Association. This self-archived article is provided for non-commercial and scholarly purposes only. Forking Paths 2 Imagine that a researcher predicts that men have higher self-esteem than women. Following data collection, the researcher observes that self-esteem scores in the sample are negatively skewed and, therefore, not ideal for a parametric test. Consequently, the researcher decides to log10 transform the selfesteem scores in order to produce a more normal distribution. Although this transformation makes the selfesteem scores more suitable for a t test, it also causes a statistical problem that increases the probability of incorrectly rejecting the null hypothesis that there is no gender difference in self-esteem (i.e., making a Type I error).This example represents part of a broader problem that Gelman and Loken (2013, 2014) have described as the garden of forking paths. Forking paths represent tests that are based on different analytical approaches, such as transforming or not transforming self-esteem scores. In the above example, the researcher followed only one analysis path (transform the scores) because it was considered to be the best (most valid) path to take. Nonetheless, the fact that a second potential analysis path exists (do not transform the scores) increases the probability of making a false positive (Type I) error during hypothesis testing.In their articles on this subject, Loken (2013, 2014) provided some useful illustrations of the forking paths problem using examples from several published research articles. However, they did not pro...