ADVISING ON RESEARCH METHODS
PROCEEDINGS OF THE 2007 KNAW COLLOQUIUM

ADVISING ON RESEARCH METHODS

Proceedings of the 2007 KNAW Colloquium

Amsterdam, March 28–30, 2007

Edited by Herman J. Adèr and Gideon J. Mellenbergh

© Herman J. Adèr and Gideon J. Mellenbergh 2008

First published 2008

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, transmitted or utilized in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without permission in writing from the Publisher.

Huizen, The Netherlands
E-mail: Publishing@jvank.nl

ISBN 97-890-79418-30-5
NUR: 840
Funded by the Royal Netherlands Academy of Arts and Sciences KNAW
Cover design: Sonja Duijkers

Preface

Researchers in the behavioural, medical, and social sciences are permanently confronted with hard methodological problems. These problems often originate from the fact that the objects of study of these sciences are human beings, who are active participants in research studies. For example, humans do not need to cooperate with researchers, may drop out of studies, may react to measurements in unexpected ways, measurement data may not comply to the assumptions of conventional statistical methods, and so on.

The last hundred years the behavioural, medical, and social sciences have attacked these problems by developing a large body of methodological knowledge. This knowledge covers philosophical insights in the foundations of research, designs for human research studies, psychometric methods for the development and analysis of measurement instruments, and statistical methods for the analysis of empirical data. Methodology has become a broad field, and specific methods have been developed in each of the different disciplines of the behavioural, medical, and social sciences. Nevertheless, the methodologies of the different fields have much in common. Therefore, we think that methodologists of different substantive disciplines need to keep contact, to exchange experiences, to cooperate and to work together on the development of new methods.

An important part of the job of most methodologists is advising substantive researchers on research methods. Methodological consultancy impresses outsiders as an art, and methodologists sometimes reinforce this impression. However, useful information is available on both the communication (see, for example Derr, 2000) and content (see, for example Adèr, Mellenbergh, & Hand, 2008) aspects of methodological consultancy, and this knowledge can be passed on to starting methodological consultants.

Our convictions that methodology cuts different disciplines and that methodological consultancy can be trained settled in Herman Adèr the plan to organize a meeting focussed on the topic of advising on research methods for methodologists from different disciplines. The Royal Netherlands Academy of Arts and Sciences (KNAW) accepted the plan, and offered to finance and support an academy colloquium on advising research methods. The colloquium was held March 29th and 30th 2007 in the KNAW-building at Amsterdam. The colloquium was preceded March 28th by a one-day KNAW masterclass Advising on Research Methods for starting methodological consultants from different substantive fields.

The colloquium covered a broad spectrum of consultancy topics. Several speakers extended their contribution to a paper which is published in the present proceedings of the colloquium¹
____________________________
¹All abstracts, handouts and presentations can be found at the KNAW colloquium website: http://www.knaw.nl/colloquia/

.

Advising on research methods in a multi-disciplinary setting is discussed by both Janice Derr and Hilde Tobi. Derr describes barriers to the effectiveness of statistical consultancy in multi-disciplinary settings, and practical ways to overcome them. Tobi takes Lattuca’s distinction between four types of interdisciplinarity as a starting point to describe the role of the methodological consultant in each of these four types (Lattuca, 2003). She also discusses the possibilities and challenges of working in interdisciplinary research settings.

Gerald van Belle links statistics to everyday experience. He considers statistics as a distillation of human experience of variation and causation. Statistical consultants can take advantage of this link by appealing to their clients’ experience of variation and causation. Victor van Daal comments on Van Belle’s views. He expands Van Belle’s topics of variation and causation by connecting them to De Groot’s phases of the empirical cycle of research (De Groot, 1969).

The contributions of both Robert Pool and Jules Ellis are on methodological consultancy in practical setting. Pool describes a clinical trial on the prevention of HIV transmission in Africa. The exclusive use of conventional surveys and questionnaires is out of the question in this context. He describes pitfalls and solutions of getting information on the sensitive topic of this clinical trial. Ellis reports consultancy to companies, which did research on the financing of the health care in The Netherlands. He discusses the effectiveness of advice given the constraints of prior ideas and requirements of decision makers and stakeholders.

The contributions of Bo Lu and Marijtje van Duijn and of Herman Adèr are more theoretical than the other contributions. Lu and Van Duijn discuss the use of propensity scores to reduce selection bias in observational studies. They present a simulation study to illustrate that propensity score matching can be effective in reducing selection bias. Their contribution shows that the propensity score should be part of every consultant’s methodological toolbox. Adèr addresses two related topics. First, the importance of the temporal order of observations and events. Second, the application of research strategies, such as structural equation modelling and propensity score matching. The two topics are related because the application of a research strategy implies a temporal order of actions, which have to be taken. His contribution alerts consultants to the importance of temporal order and the application of ordered actions in a research strategy.

We thank the board of the KNAW for giving financial support and affording the opportunity to organize the meetings in the historical KNAW-building ‘Het Trippenhuis’. We thank KNAW-board member Jacqueline Meulman for her cordial and informed introduction to the colloquium, and KNAW-office members Martine Wagenaar and Sonja Duykers for their pleasant and competent assistance. Finally, we thank the speakers, discussants, and other participants for their interesting contributions to the field of methodological consultancy.

July 7, 2008

Gideon J. Mellenbergh, Amsterdam

Herman J. Adèr, Huizen

References

   Adèr, H. J., Mellenbergh, G. J., & Hand, D. J. (2008). Advising on research methods: A consultant’s companion. Huizen: Johannes van Kessel. (With contributions by David Hand)

   De Groot, A. D. (1969). Methodology. The Hague, The Netherlands: Mouton.

   Derr, J. A.  (2000). Statistical Consulting: A guide to effective communication. Pacific Grove, CA: Duxbury Press.

   Lattuca, L. R. (2003). Creating interdisciplinarity: Grounded definitions from college and university faculty. History of Intellectual Culture, 3(1). Retrieved from http://www.ucalgary.ca/hic