ADVISING ON RESEARCH METHODS
SELECTED TOPICS 2012

Advising on research methods

Selected topics 2012

Edited by

Herman J. Adèr and Gideon J. Mellenbergh

Johannes van Kessel Publishing
Huizen, The Netherlands

© Herman J. Adèr and Gideon J. Mellenbergh 2012

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, transmitted or utilized in any form or by any means, electoronic, mechanical, photocopying, recording or otherwise, without permission in writing from the Publisher.

Huizen, The Netherlands
E-mail: Publishing@jvank.nl

ISBN 978-90-79418-28-2

NUR: 840, General Education

Preface

Herman Adèr and I teach the research master course Methodological Advice of the Department of Psychology, University of Amsterdam. The objectives of this course are: (1) to acquire methodological knowledge that is needed for advising researchers in the behavioral and social sciences, and (2) to get experience with methodological consultancy.

The students of the course get a number of different assignments. In 2011 we started to give students the assignment to write a paper on a topic that occurs in methological consultancy. The students were instructed to write a paper that had to be published in a book. The intended audience of the book were fellow research master students who gave methodological advice on research in the behavioral sciences, such as is done in Methodology Shops of Dutch psychological departments. The procedure to prepare the book resembles the procedure that is used to prepare an edited book. The authors wrote a first draft of their article, this draft was reviewed by other students and the course instructors, and the authors used the comments to rewrite the first drafts. The assignment appeared to be a success. The students found it a hard job, but they appreciated this learning experience. Their work resulted in the book:

Advising on research methods: Selected topics 2011
edited by H. J. Adèr and G. J. Mellenbergh, and
published by Johannes van Kessel in 2011.

The same assignment was used in the 2012 course. The students selected a topic from a list of topics, and wrote an article on this topic. This was done either in pairs or individually. When writing in a pair, the authors of the article are equally responsibler and their names are reported in alphabetical order in the book. Each of the other students also selected a topic from the list, and each of them wrote an article on the selected topic.

The students made the following contributions to the book:

Jonas Dalege and Lous Kreemers address the topic of measurement invariance with respect to a grouping variable (e.g., gender, experimentally manipulated groups). They discuss multi-group factoranalytic models to detect measurement invariance with respect to a grouping variable. Moreover, they present steps the consultant has to take when he or she is advising a client on a measurement invariance problem.

Marie K. Deserno introduces consultants to the two main types of models for the measurement of psychological and educational constructs, that is, Classical Test Theory (CTT) and Item Response Theory (IRT). She focuses on the differences between CTT and IRT. She concludes by pointing consultants to two problems that cannot be solved under CTT: Computerized Adaptive Testing and Differential Item Functioning, which is a special case of measurement invariance, can only be handled under IRT.

Milou K. M. Lünnemann discusses unit nonresponse in surveys. She defines nonresponse and describes the bias of nonresponse in survey studies. She ends her article with guidelines to prevent nonresponse in the design and implementation of survey studies, and to correct for nonresponse in the analysis of the data of survey studies.

Adam Sasiadek and David Scholz discuss the application of effect size measures in meta-analysis. They distinguish two ways of founding effect size measures, that is, measures based on means and measures based on correlations. Finally, they give guidelines for the application of effect size measures in meta-analysis.

Suzan Q. Blommestijn and Esther A. Lietaert Peerbolte  
discuss outliers and extreme observations in empirical data. They explain the concepts of outlier and extreme observation. Moreover, they describe methods to detect outliers in univariate and bivariate distributions. Finally, they mention steps that the consultant can take when he or she advises clients on the handling of outliers.

Jochem Bout discusses questionable practices in empirical research. He describes some strategies to counteract these practices. Finally, he mentions guidelines for consultants that can be used to prevent clients to apply questionable research practices.

The authors worked under great time pressure because the whole procedure of writing, reviewing, and rewriting had to be done within one month. Nevertheless, they managed to make interesting and relevant contributions to the book.

We thank the authors for their enthusiasm, competence, and working speed. Moreover, we thank Johannes van Kessel Publishing for producing this book and for producing it within fourteen days. We think this Selected Topics 2012 issue of Advising on Research Methods, as well as the 2011 issue, is useful for methodological adivisors, and we hope that it will be used in methodological consultancy.

Amsterdam, December 2012

Gideon J. Mellenbergh
Herman J. Adèr
(course instructors)

Chapter 6
Questionable research practices and scientific fraud

by Jochem Bout

Scientific fraud forms a danger for the integrity of science. It is clear that fabrication, falsification or plagiarism have to be dealt with. But for the more frequently used questionable research practices the dangers are not so clear. Examples of questionable research practices are “collecting more data after seeing whether results were significant” and “claiming to have predicted an unexpected finding”. This chapter discusses these kind of questionable research practices.

Keywords: Scientific fraud, sloppy science, Diederik Stapel, Levelt report, scientific integrity, HARKing, small data set, data documentation, analysis documentation, reporting, meta analysis, publication bias, file-drawer effect.

6.1 Introduction

Scientific fraud and questionable research practices are currently a hot topic in the media. The committee that investigated the data fabrication of Stapel concluded that there are currently some bad practices in the psychological research fields. There is a verification bias and studies are not frequently replicated. The Levelt report calls this “sloppy science” (2012).

Scientific fraud forms a danger for the integrity of science. It is clear that fabrication, falsification or plagiarism have to be dealt with. But for the more frequently used questionable research practices the dangers are not so clear. Examples of questionable research practices are “collecting more data after seeing whether results were significant” and “claiming to have predicted an unexpected finding”. These kind of practices are dangerous especially when used in combination with small studies. They also form a danger for meta-analyzes because they bias the effect size. One of the reasons for researchers to use questionable research practices is to get positive results. This is important for researchers because there is a publication bias for positive significant results. Publication bias creates a file-drawer effect. To decrease this effect it is important to make researchers, editors and sponsors aware of the bias for positive results. Another solution is to obligate researchers to put all their research in an open database so that it is all accessible. From the consultant’s point of view it is important to make researchers understand the importance of reporting each step in the analysis and the obtaining of data.

There is a distinction between scientific fraud and questionable research practices. The definition of fraud is fabrication, falsification, or plagiarism in proposing, performing, or reviewing research, or in reporting research results. (OSTP, 2005, cited in Martinson, Anderson, & de Vries, 2005). In the case of questionable research practices it is not always clear to researchers whether they may be harmful. This is why they are more frequently used and therefore form a greater danger to the integrity of science than ‘straightforward’ fraud Martinson et al.. This is why the emphasis in this chapter will be mainly on questionable research practices.

The first part of this chapter exists of an overview of different forms of questionable research practices and fraud. The second part discusses the dangers of these practices. The third part will explain why scientists turn to such practices and in the fourth part we will discuss some solutions and practical tips for methodological advice.

6.2 Overview

There are a lot of different questionable research practices. In some cases researchers do not know that what they are doing, is wrong. For example, using inappropriate statistical methods to enhance the significance of findings (Piantadosi, 1997). In other cases researchers consciously make use of questionable research practices. For example, hypothesizing after the results are known (Kerr, 1998). Table 6.1 describes some of these questionable research practices, mentioned by (John, Loewenstein, & Prelec, 2012). They investigated 2000 psychologists with a survey about (percieved) fraud and questionable research practices.

Table 6.1: Questionable research practices (John et al., 2012).
(Percentages are by approximation).

6.2.1 HARKing

One questionable research practice mentioned in Table 6.1 is “claiming to have predicted an unexpected finding”. This is called Hypothesizing After the Results are Known, also known as HARKing (Kerr, 1998). There are a few forms of HARKing. When researchers use ‘Pure HARKing’ they make a post-hoc hypothesis that is consistent with the data and present it as if it was a priori. Another approach is Pure HARKing + Straw Man. This means that a researcher presents a hypothesis that is consistent with the data and adds an hypothesis that is contradicted by the data. In this case the hypotheses are also presented as if they were a priori. Another form of HARKing is to suppress so-called loser hypotheses. So the researcher only presents the hypothesis that are consistent with the data.

There are a few problems that are caused by HARKing. It breeds cynism about science because it violates principles that good science is based upon. Furthermore it can have a negative effect on scientific discoveries because researchers focus on the post-hoc hypothesis that are presented as a priori. Therefore researchers ignore possible alternative hypothesis (Kerr).

It may be clear that HARKing is a problem. Now there will be discussed some solutions to HARKing. It is important to make researchers and students familiar with the dangers of HARKing. It is also important to obligate researchers to make a research proposal. If this is done it is clear what the a priori hypotheses are and how they are tested. It is possible for journals to refuse an article if this requirement is not met (Kerr).

EXAMPLE 6.1 (A typical case of HARKing).

In research by Bem, Utts, and Johnson (2011), it was suggested that future events retroactively affect people’s responses. This finding is very controversial given current theories of behavior in psychology. This is why Wagenmakers, Wetzels, Borsboom, Kievit, and van der Maas (2011) published a critical review of Bem et al.’s article. They found that the authors had done exploratory analyzes in their studies and reported them as if they were confirmatory. They also found that some of the data in the studies was used twice. First to form a hypothesis and then to test and confirm it. This is a typical case of HARKing. They also redid the analyzes using Bayesian methods and showed that the evidence was weak to nonexistent.

6.3 The dangers of questionable research practices and fraud

It is clear that fraud in the form of falsifying data and plagiarism damages the integrity of science. But the influence of questionable research practices is not so clear. We will discuss some effects of the use of questionable research practices. First, we will discuss the effects on false positives, also called type I errors, and the influence on effect sizes in meta-analyzes.

Bakker, van Dijk, and Wicherts (2012) did a simulation study about the influence of questionable research practices on the bias of the effect size in meta-analyzes and the influence on false positives. The bias was the difference between the average estimated effect size and the true effect size. They tested four strategies. In the first strategy they used one large study with sufficient power. The second strategy consisted of one large study with the use of questionable research practices. The third strategy consisted of five small studies in which the data collection was stopped when a significant result was found and only the studies with a significant result were reported. The fourth strategy consisted of the third strategy with the use of questionable research practices. They found that the third and fourth strategy leads to the most bias in effect size. They also found that the fourth strategy leads to an inflated type I error up to 40%.

This study shows that it is dangerous to use several small sample studies and only report the ones that have significant results. Underpowered studies bias effect sizes in meta-analyzes and inflate type I errors.

6.3.1 Why do researchers turn to such practices?

One of the reasons that researchers make use of questionable research practices is to find significant results. There is a reason why researchers focus on significant results. This is because there is a tendency by researchers, journal editors and research sponsors to focus on ‘interesting’ results. This means that they focus on results that are significant in the way it was hypothesized. This tendency is called publication bias and causes research to disappear in the file-drawer. There is evidence that researchers are less likely to report and submit negative results (Gilbody, Song, Eastwood, & Sutton, 2000). But even if a researcher submits negative results it is less likely that a journal editor publishes the article. The kind of sponsor is also an influential factor for publication. If research is sponsored by a commercially sponsor the article is less likely to be published if the results are negative. With public sponsors an article is likely to be published regardless of the results.

6.4 Solutions

In this part of the chapter we will discuss some solutions for minimizing the use of questionable research practices.

Rules. There are currently no clear rules concerning the use of questionable research practices and the use of multiple small studies Bakker et al. (2012). Therefore it is important to make documentation concerning the use of small studies, reporting standards, the use of nonsignificant results and the difference between exploratory and confirmatory studies. When these rules are clear it is important to sanction researchers who did not follow them. One sanction could be that journals will not publish articles that did not clearly followed the rules.

Open database. With the technology nowadays it is easy to make your results and data accessible to everyone. Even if there is a bias towards positive significant results it is important that studies with nonsignificant or negative results are also available, especially for meta-analyzes. For psychological research there is www.psychfiledrawer.org. This is a website that gives researchers the possibility to make replication of experimental psychological studies accessible to the rest of the world. This is a positive development. But it would be great if the whole scientific community would endeavor to more openness. Maybe it is time to change the prevailing publication system with journals and editors. It is expensive for universities to get a subscription to those journals. If universities would invest this money in a collaborative project to make an open database, with clear rules, in which researchers are obligated to publish their research, including data and research proposal, science would be more open. Of course there has to be some supervision. This can be done by assigning other scientists that use the database to comment on the article.

Table 6.2: Consultant’s tips to the client.

6.5 Methodological advice

Consultants can contribute to the improvement of research practices by making clients aware of the questionable research practices discussed above. It is also important for consultants to advice clients to document and report every step in the data collection and analysis of the data. Although this is time consuming for clients it is the only way for other scientists to correct unintentional mistakes and replicate a study. It also contributes to preventing other researchers from making the same mistakes. Table 6.2 gives an overview of few tips consultants can give to clients. They are based on an article by Simmons, Nelson, and Simonsohn (2011). Furthermore, consultants can make clients aware of open databases like www.psychfiledrawer.org.

6.6 Discussion

Questionable research practices and fraud form a danger for the integrity of science. They bias effect sizes and inflate type I errors. Publication bias lure researchers into focussing on positive significant results. Scientists should be aware of this tendency. Solutions to minimize the use of questionable research practices include reporting standards and bias due to small study size. Scientists should also be encouraged to use open databases to minimize the effect of publication bias. For a consultant it is important to stimulated their client to develop clear reporting habits about data gathering and data analysis.

It will take a lot of time and effort to cleanup the “sloppy science” mentioned in the Levelt report report, hut if the scientific community is dedicated to doing this cleanup we will only talk about “neat science” in the future.

References

   Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called Psychological Science. Perspectives on Psychological Science, 7(6), 543–554.

   Bem, D. J., Utts, J., & Johnson, W. O. (2011). Must psychologists change the way they analyze their data? A response to Wagenmakers, Wetzels, Borsboom, & van der Maas (2011) . (Submitted for publication)

   Gilbody, S. M., Song, F., Eastwood, A. J., & Sutton, A.  (2000). The causes, consequences and detection of publication bias in psychiatry. Acta Psychiatr Scand, 102, 241–249.

   John, L. K., Loewenstein, G., & Prelec, D.  (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532.

   Kerr, N. L.  (1998). HARKing: Hypothesizing After the Results are Known. Personality and Social Psychology Review, 2(3), 196–217.

   Levelt Committee, Noort Committee, & Drenth Committee.  (2012). Flawed science: The fraudulent research practices of social psychologist diederik Stapel (Tech. Rep.).

   Martinson, B. C., Anderson, M. S., & de Vries, R.  (2005). Scientists behaving badly. Nature, 435, 737–738.

   OSTP Federal Policy on Research Misconduct. (2005). Published online. Available from www.ostp.gov/html/001207_3.html

   Piantadosi, S. (1997). Clinical trials: A methodologic perspective. New York: Wiley.

   Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-Positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 1–8. Online article.

   Wagenmakers, E., Wetzels, R., Borsboom, D., Kievit, R., & van der Maas, H. L. J. (2011). Yes, psychologists must change the way they analyze their data: Clarifications for Bem, Utts, and Johnson (2011).