Fact or Fluke?

A Critical Look at Statistical Evidence

Ronald Meester and Klaas Slooten

Amsterdam University Press

Also published in Dutch as: Kan dat geen toeval zijn? Een kritische blik op statistische bewijsvoering, Ronald Meester en Klaas Slooten. Amsterdam University Press, 2022. DOI 10.5117/9789463725088

Translation: Reinie Erné

Cover design: Gijs Mathijs Klunder

Lay-out: Crius Group, Hulshout

ISBN

978 94 6372 349 7

e-ISBN

978 90 4855 744 8

DOI

10.5117/9789463723497

NUR

916

© R. Meester & K. Slooten / Amsterdam University Press B.V., Amsterdam 2022

All rights reserved. Without limiting the rights under copyright reserved above, no part of this book may be reproduced, stored in or introduced into a retrieval system, or transmitted, in any form or by any means (electronic, mechanical, photocopying, recording or otherwise) without the written permission of both the copyright owner and the author of the book.

Every effort has been made to obtain permission to use all copyrighted illustrations reproduced in this book. Nonetheless, whosoever believes to have rights to this material is advised to contact the publisher.

Table of Contents

Preface

Prologue

Part I Classical Statistics

1. Significance Testing

1.1 Testing the Null Hypothesis and Statistical Significance

1.2 The Logic of Significance Testing: In the Words of Fisher

1.3 Significance Testing Ignores the Context

1.4 Back to Sally Clark

2. p-Values

2.1 What Is a p-value?

2.2 The Main Problem with p-Values

2.3 Publication Bias

2.4 One-Tailed Versus Two-Tailed: A Paradox

2.5 The p-Value in Adaptive Sampling Studies

2.6 More on Adaptive Sampling Studies

3. Confidence Intervals

3.1 What Is a Confidence Interval?

3.2 Confidence Intervals, p-Values, and Effect Size

3.3 Dependence on the Experimental Setup

3.4 Strange (and Amusing) Confidence Intervals

Part II A Bayesian Approach

4. What Is Statistical Evidence?

4.1 The Likelihood Ratio

4.2 Likelihood Ratios for an Unknown Probability of Success

4.3 The Likelihood Ratio Solves Problems with p-Values

4.4 The Interpretation of the Likelihood Ratio

4.5 p-Values versus Likelihood Ratios

4.6 Likelihood Ratios and Power

5. Evidence and Belief

5.1 Alternative Hypotheses and Context

5.2 A Return to Ioannidis’ Argument

5.3 An Anecdotal Cards Example

5.4 A Philosophical Interlude

5.5 Worked-Out Examples – Credibility Intervals

5.6 Laypersons and the Prior

5.7 Objective Bayes?

5.8 A Few Conclusions

6. The Likelihood Ratio and the Experimental Setup

6.1 Error Probabilities and Misleading Evidence

6.2 How Often Does Misleading Evidence Occur?

6.3 Likelihood Ratios and Designing an Experimental Setup

6.4 Conclusions

Part III Statistics in Practice

7. Two Worked-Out Examples

7.1 Face Masks

7.2 The Lucia de Berk Case

8. Sometimes p-Values Can Be Justified

8.1 Elementary Particles in Theoretical Physics

8.2 Model Validation

Appendix

Bibliography

Index

Preface

This book is intended for anyone who is in some way interested in statistical evidence: scientific researchers, students, teachers, mathematicians, philosophers, lawyers, managers, and probably many others. It is not an ordinary book about statistics, and it certainly is not a recipe book that tells the reader which test goes with, or which software can be used, for which problem. So what kind of book is it? We will try to explain that now.

In this book we explain how statistical argumentation works. We will see what types of questions are asked, and what logic underlies how we try to answer those questions. Broadly speaking, there are two ways of doing statistical argumentation, both of which have a long history. One way is to consider a research hypothesis proved if the obtained data fit very poorly with the research hypothesis not being true. The other way consists of looking for the best explanation for the data one wants to interpret. These forms of statistics are known as, respectively, the classical (or frequentist) approach and the Bayesian approach. Contrary to what the names suggest, the Bayesian approach has the oldest foundations: it goes back to Thomas Bayes (1702–1761), whereas the frequentist approach was developed mainly from the beginning of the twentieth century onward.

There are many textbooks in statistics that explain how, in the classical approach, to interpret data, formulate hypotheses, design experiments, and decide whether to reject a hypothesis. In this book, we want to show that certain aspects of this classical interpretation of evidence are problematic. As a result, reported claims may be unfounded. These problems with statistical evidence are partly responsible for what is being called the replication crisis, the apparent inability to reproduce research results. In other words, it is sometimes impossible to confirm the conclusion derived from a first experiment through a second experiment. What was considered “proved” scientifically may, on closer inspection, not be so.

This is of course a serious problem. The reliability and status of statistical evidence are at stake. This may have far-reaching consequences, not only for science but also for society as a whole, for example for health care or the administration of justice. After all, data are gathered not only to investigate whether a particular hypothesis is correct, but also to draw conclusions or attach consequences from its correctness (or incorrectness). This can happen in all sorts of subject areas, because the data evaluated statistically can relate to anything. A statistical interpretation is needed whenever these data are (partly) determined by chance. This may concern the effectiveness of a medicine, the relationship between diet and health, opinion polls, or elementary particles colliding in a particle accelerator; in each case, the collected data are determined partly by chance.

In this book, we will explain, hopefully in a readable and understandable way, how it is possible that statistics is fraught with so many problems. We will have to consider many things in doing so. What is statistical evidence anyway? This may seem like a simple question, but we will see that it is much more complex than one might think at first glance. We will therefore try to go back to the essence, namely the way of thinking and the reasoning behind statistics, because they show us what we can and cannot expect from statistics. Using this approach, we will explain that the classical way of testing hypotheses does not answer the question whether the data provide evidence for or against a hypothesis and is therefore an inadequate method for drawing conclusions about it.

Next, we will try to show how thinking about the nature of statistical evidence can put us on the right track to reason in a different way and arrive at conclusions that are logically justified.

We are of course not the first to criticize the current statistical practice, and some of the problems identified in this book have already been recognized by colleagues. Let us mention a few. Biostatistician Richard Royall already voiced strong criticism of classical statistics in 1996 [27]. In 2005, epidemiologist John P.A. Ioannidis published an article in which he stated that most published scientific research results are false [18]. Recently, statistician William Briggs, for example, also rejected the use of the classical p-values [6]. There are also journals that have banned the use of the classical p-values, such as Basic and Applied Psychology, Epidemiology, and Political Analysis; see, for example, [10], [15], and [30]. That little has changed so far despite these—and many similar—publications is remarkable, especially when one considers that solutions have also been proposed, about which we will say (much) more later.

Our contribution differs from the previously mentioned publications in that we do not focus primarily on mathematics but mainly explain how classical statistics works and why it works that way, why and when it sometimes fails, and what one could do instead. We want to show that statistics as we advocate it is very much in line with our intuition about what we can expect from statistical research. We explain what we can or may, and also what we cannot or may not, expect from statistical evidence, to what extent objectivity can be found or pursued, where objectivity ends, and whether it is an advantage or disadvantage for a method to be objective. Each time, we explain the reason for our approach to the reader and try to point out the problems. This book therefore is both philosophical and applied. That makes it quite unique: we have not found any other texts in the literature with this approach, even though we believe that it is the best way to understand the nature of statistical reasoning. We therefore hope that this book will contribute to a more realistic view of statistical evidence.

The book is structured as follows. In Part I, we discuss classical statistics. The first chapter deals with significance testing. We first conclude that it is possible that many reported claims may be false, and we give a chilling example from legal practice of what can go wrong. In Chapter 2, p-values are discussed in detail. We explain what a p-value is and why it cannot be adequately used to measure the strength of statistical evidence. If, as often happens, p-values are interpreted as such, this can lead to problems and errors. Chapter 3 deals with confidence intervals. These are closely related to p-values and are subject to similar interpretation problems, which we also discuss extensively.

In Part II, we discuss a more Bayesian approach. We begin, in Chapter 4, by explaining what statistical evidence truly is, and we introduce an instrument to express evidential strength: the likelihood ratio. We conclude that statistical evidence can seldom be absolute. Instead, it is usually relative to an alternative explanation. In Chapter 5, we give several concrete examples of likelihood ratios in action and discuss the critical difference between evidence and belief. Next, in Chapter 6, we consider the extent to which data can be misleading. After all, it is unreasonable to expect every shred of data to lead to the truth about how the data came to be, but one can ask how often and to what degree it can go wrong. Fortunately, there are limits to this. We also show how the likelihood ratio can be used to design experimental setups. This concludes Part II.

Part III is devoted to statistics in practice. In Chapter 7, we present a comprehensive statistical analysis of two concrete situations. In Chapter 8, we explain that there are situations in which p-values can be used pragmatically despite the methodological problems associated with them.

Mathematically, the book does not surpass the level of a first course in applied statistics. The most complicated mathematics consists of calculating some integrals, using conditional probabilities, and applying the quadratic formula.

However, the mathematical content is often not the biggest problem in understanding statistics. Statistics is more than mathematics. The difficulty lies mainly in the meaning of what we do. So although we try to present the material in the most accessible way possible, some effort is required of the reader. There may be sections that the reader may need to review several times, and it may be necessary to use pen and paper to go over our calculations. We think the reward is more than worth it.

The reader familiar with mathematical statistics may be surprised to learn that the concepts propagated in this book have long been accepted within mathematical statistics. What we hope to achieve, however, is that a reflection on what statistics can or cannot do will almost automatically lead to a more or less Bayesian approach without losing sight of the valuable aspects of the more classical “frequentist” statistics.

Some sections in this book are a bit more technical and are labeled with an asterisk (*). This means that the content of these sections is a bit more complex and can safely be skipped—though we do of course recommend them to enthusiasts. We hope the reader enjoys this book and that reading it will be worth the effort.

Several people have helped us immensely with all sorts of comments on earlier versions of this manuscript. Our thanks go to (in alphabetical order) Marc Jacobs, Wouter Kager, Boukje Meester, Luit Jan Slooten, Robert van der Toorn, Wessel van Wieringen, and Harry van Zanten.

Ronald Meester and Klaas Slooten

Prologue

In December 1996, young Christopher, the first son of the British solicitor Sally Clark and her husband, was found lifeless in his bed. At the time of his death, Christopher, not yet three months old, was home alone with his mother. His death was not found suspicious and was classified as sudden infant death syndrome (SIDS).

In January 1998, something similar happened to Sally Clark’s second son Harry. While alone with his mother, Harry, who was almost two months old, suddenly and inexplicably became unwell and died shortly after. Doubts now arose about the cause of his death as well as Christopher’s. Shortly thereafter, Sally Clark was charged with the murder of both of her sons.

An important role in the trial was played by a witness called by the prosecution, Professor of Pediatrics Sir Roy Meadow. According to Meadow, the probability of two consecutive children within a family both dying of SIDS was about 1 in 73 million. This number was questionable: it was obtained by squaring the postulated (and in itself already questionable) probability of an individual SIDS death, 1 in 8,543, because it concerned two children. This assumes that occurrences of SIDS in different children from the same family are independent of one another. This is questionable because (common) genetic aspects may play a role, as can certain aspects of the children’s care.

But the fact that the number of 1 in 73 million was questionable is not our main concern; undoubtedly, the probability of two SIDS deaths in two consecutive children is small. A much greater problem is in the following reasoning, which became tempting for the jury: if Sally Clark were innocent, something extremely unusual would have to have occurred, namely an event with probability 1 in 73 million. Sally Clark was found guilty on November 19, 1999 by a 10-2 majority verdict of the jury and sentenced to life imprisonment for two counts of murder. The conviction was upheld at appeal in 2000. Meanwhile, statisticians began to get involved in the case: in 2001, the Royal Statistical Society raised concerns about both the basis for Meadow’s number and its application in the case against Sally Clark. In 2003, the conviction was overturned, and Sally Clark was released from prison.

How could this happen? How can something that first seemed to be strong statistical evidence turn out not to be so? Why is Meadow’s number not proof that Sally Clark killed her children? And how does one actually determine what statistical evidence is and how strong it is? These are all questions we address in this book.

Part I

Classical Statistics

1.Significance Testing

In 2005, the article “Why Most Published Research Findings Are False,” written by epidemiologist John Ioannidis, appeared in PLOS Medicine [15]. In this article the author argued that it should not be surprising that most statistically proved connections do not truly exist. This is not so much due to errors in the statistical calculations, but rather to a misunderstanding of what conclusions can be drawn from the results of those statistical analyses. The article hit like a bombshell, judging by the many thousands of times it has been cited. For it immediately provides an explanation for what has become known as the “replication crisis,” namely that a second group of researchers often fails to obtain the same research result as a first group. This of course raises the question whether that result was correct. In some disciplines, this fate struck so many of the published research results that there was a genuine crisis: somehow, a significant portion of the seemingly proven results seemed (or turned out) to not be true (see, for example, [4] for the results of a survey on this by Nature). This is explained precisely by Ioannidis’ article. His argumentation provides a simplified representation of reality, yet he manages to expose a problem that is indeed real using elementary arguments. That is why Ionnadis’ approach is a good beginning for this book.

1.1Testing the Null Hypothesis and Statistical Significance

In his article, Ioannidis assumes that conclusions are drawn based on so-called significance testing, often referred to as NHST (null hypothesis significance testing). This type of testing follows a fixed pattern, which, at first glance, seems entirely reasonable and logical. In this chapter, we explain in detail what this type of testing involves, why it is designed as it is, and what disadvantages it has when widely applied to a variety of hypotheses. Simply put, Ioannidis describes what would happen if careless researchers applied this decision scheme on a large scale and made decisions based solely on the resulting statistical analysis. Of course, this is not always how things are in real life: the quality of a statistical analysis and the subsequent decision-making involve much more than what Ioannidis takes into account. But as a global, qualitative analysis, Ioannidis’ reasoning is absolutely valuable. The testing protocol he describes is indeed one of the most widely used in statistics. This approach was designed over a century ago by famous statisticians such as Fisher, Neyman, and Pearson. A description of this approach is therefore not complete without examining the motives, expectations, and interpretation of the results inspiring this paradigm.

1.1.1The Null Hypothesis

Let us begin by explaining what significance testing entails. Suppose that researchers are looking for evidence for a particular hypothesis, for example the hypothesis that there is a relation between a specific gene and developing a particular disease, or the hypothesis that a new medicine works better than an old one. To do scientific research, the researchers establish a procedure helping them to ultimately draw conclusions. In the first example, the researchers may check a large number of subjects to see whether they have certain DNA variants on the specific gene and whether they get the disease in question. In the second example, the researchers may proceed as follows. Some patients are given the new medicine, while others are given the old one, in such a way that neither the patients nor the doctor treating them know who is getting the new medicine and who is not. The different recovery rates in the two groups can then be used to determine whether the new medicine works better than the old one.

Of course, this is not an accurate description of any research. In the medicine example one can think of all sorts of rules to avoid using the old medicine when it has become clear that the new medicine is indeed better. This complicates the analysis, but the philosophy remains the same, which is what we want to discuss now. Let us, in addition, note that we are speaking purely of statistical argumentation. There may of course be physiological or biological reasons to study that specific gene, and there are probably also biochemical reasons why the researchers expect, hope, or think that the new medicine works better than the old one. However, the evidence these experiments will or will not produce is statistical and must therefore be interpreted statistically.

So, how does such an experiment go? Well, in the simplest but most common case (which is what we assume here), the researchers formulate a so-called hypothesis H called the null hypothesis. Researchers often formulate H as the hypothesis they would like to reject, in the sense that they think or hope that this hypothesis is incorrect. In the first example, H may be the hypothesis that the gene has no link to the disease’s occurrence, while in the second case, H may be the hypothesis that the new medicine does not work better than the old one. The hypothesis the researchers would like to confirm is therefore the negation of H; this is also called the research hypothesis. The negation of the null hypothesis is often not very specific: whereas the null hypothesis says that there is no effect, its negation says that there is an effect, without specifying how large it is. We will discuss the implications of this in detail later.

Next, the researchers determine in advance for which research outcomes they will reject the hypothesis H and for which outcomes they will not reject it. If H is rejected, this is generally interpreted as meaning that the obtained data provide evidence against H and therefore for the negation of H. In the first example, rejecting H is taken to mean that there is a correlation between the gene and the disease, and in the second example that the new medicine is better than the old one. Of course, it is also possible that the data do not lead to rejecting the null hypothesis. This is then interpreted as meaning that the data do not provide (sufficient) evidence for the incorrectness of the null hypothesis.

Of course, the researchers must give much thought to when they want to reject H, and it is precisely in that choice that probability and statistics enter the picture. At the end of the procedure, the researchers may reject the hypothesis even though it is true. Conversely, the researchers may not reject the hypothesis H even though it is in fact false. In other words, the researchers may draw an incorrect conclusion, and needless to say, we need to consider the probability of that happening.

1.1.2Significance and Power

In the classical procedure (which we describe here), this roughly goes as follows. The researchers would like to reject the null hypothesis H (because they think it is false), but they do not want to draw conclusions lightly, so they want some control over the probability of wrongly rejecting it. After all, it would be very problematic if research hypotheses were too often considered proved when they are not true. So, the researchers will require the probability of wrongly rejecting H to be very small. This requirement implies that the researchers must be able to say something about the probabilities of the possible outcomes of the experiment under the assumption that H is correct.

That all seems legitimate and reasonable. Of course, it is not entirely clear what exactly “very small” means here. This seems to depend a bit on the field, but usually 0.05 or sometimes 0.01 is chosen, and even much smaller numbers may be chosen for some applications. This way, the researchers have done something important: they have ensured that they will not reject H too easily and will not conclude too lightly that the new medicine is better than the old one or that the gene is related to a particular disease. Indeed, the probability of rejecting H (the hypothesis that the new medicine is not better than the old one) even though it is true is controlled by the choice of the margin of error of, say, 0.01 or 0.05. This margin of error is usually denoted by α and called the significance level. For example, if one chooses 0.05 for α, then there is a probability of 1 in 20 that if the null hypothesis is (perhaps unexpectedly) true, the outcome will still be statistically significant.

If H is rejected at significance level α, then the study’s outcome is called significant for this α. Note that it is the study’s outcome that is significant or not, and not the claim or conclusion, if any, associated with it. The researchers make a claim or draw a conclusion on the basis of a significant outcome. Therefore, we will not speak of a “significant claim” or “significant conclusion,” only of a significant research outcome.

We are not done yet, because the researchers can also make the opposite error. They may not reject H even though it is not true. Ioannidis uses the symbol β for the probability of this happening, as is customary in the literature. The difference 1 – β is called the procedure’s power: it is the probability of rejecting H when it is indeed not true.

At the moment, it is not clear how the researchers go about designing a procedure that yields the probabilities α and β. For now, it is just a black box; we will say more about it later. Here, we are purely interested in the concept that these probabilities exist.

Let us take a moment to consider the probabilities α and β. It is important to realize that they do not have the same status. In practice, the researchers ensure that they design a procedure in which they can calculate or at least estimate α. They ensure that they reject H with probability (at most) α, assuming that H is true. Researchers will insist on a small α because they want to be sure not to be too careless in drawing a conclusion. A small α is therefore necessary; as noted above, it is generally taken to be 0.05 or 0.01. It is then customary to find, for a given α, a procedure for which β becomes as small as possible. The probability α is seen as the strength of the evidence that H is not true, whereas the power says something about the probability that an effect is observed where one exists.

We will say more about the philosophical background of these choices and of the entire procedure later on. Let us first take a look at how this all plays out in scientific practice. One might at first be inclined to think that a small probability of rejecting a true hypothesis implies that when a null hypothesis has been rejected, the probability that it was true is small. It is greatly disappointing that this is not necessarily true and even that nothing can be said about this probability without more knowledge about the context. That is the gist of Ioannidis’ story, and we will now explain it.

1.1.3Significance Alone Is Not Enough

Following Ioannidis, we assume that the probability that a hypothesis being tested is correct depends on the scientific field. When introducing a new medicine, there will often be a biochemical reason why this medicine might be better than the old one. It must of course be tested for efficacy and unwanted side effects, but once the test phase has been reached, the new medicine will frequently turn out indeed to be an improvement.

This need not be true for links between genes and a specific disease. People have thousands of genes, and one can indiscriminately test many genes at once for relevance to the disease, in the hope that the genes that matter will be picked up by the statistics. Consequently, many genes (the vast majority) are tested for which a relation to the disease does not exist. In other words, what proportion of the tested hypotheses is true varies widely from one field to another.

Let us take an example for illustration. Assume that α = 0.05 and β = 0.10 and that we are studying 1,000 research hypotheses, of which 90% are not true. We can now easily check how many of these hypotheses will lead to a significant outcome. In 900 cases, the null hypothesis H (the negation of the research hypothesis) is true, and in an expected 900 × 0.05 = 45 of these, H will be rejected wrongly because a statistically significant outcome occurs by chance. In the other 100 cases, the null hypothesis H is not true, and we expect a significant outcome in 100 × 0.90 = 90 of those cases as well. We have arranged these outcomes in Table 1.1.

Table 1.1

We see that among the 1,000 experiments, we expect a significant outcome 135 times, but only 90 of those 135 outcomes are obtained while the research hypothesis is true. The probability that the research hypothesis is false even though a significant outcome is obtained is therefore 45/135 = 1/3. In other words, in this example, if a research group obtains a statistically significant outcome, then to draw the conclusion that the research hypothesis is true would be incorrect in one out of every three cases.

That is a bit of a shock. In this example, one out of every three statistically significant outcomes is so by chance: the research hypothesis is not true after all. At first glance, the researchers seemed to be proceeding carefully by keeping the probability of wrongly rejecting a null hypothesis H under control and small, in this case at 0.05. But it now turns out that, nevertheless, the probability that a result concluded on the basis of a significant outcome is incorrect can be much greater, for example 1/3 in this case.

Things improve if more research hypotheses are true. Suppose, for example, that we turn things around and that of the 1,000 studied research hypotheses, 900 are true, and 100 are not. For the same α = 0.05 and β = 0.10, this gives Table 1.2.

Table 1.2

In this case, we expect 815 significant outcomes, of which 810 correspond to a true research hypothesis, so over 99%. This is much better than in the first example. But we do see that there is a serious problem with this hypothesis testing approach. Apparently, one cannot tell from the statistical significance of an outcome whether the research hypothesis is true, and not even the probability of it being true.

If statistical significance is seen as proof of the research hypothesis, this leads to serious problems. If most research hypotheses are correct, as in Table 1.2, then the problems are limited. Even though the conclusion that the research hypothesis has been proved is logically incorrect, it is most likely factually correct. But in a situation as referred to in Table 1.1, things really can go wrong.

By construction, the researchers have ensured that the probability of a statistically significant outcome when H is true is at most α. But this cannot be turned around: the probability that H is true when one obtains a significant outcome is different. To determine that, one needs more than α, as shown in the tables above. The problem is not in the significance testing itself but in the conclusions attached to it concerning the research hypothesis: the problem is that the experiment’s outcome does not allow one to answer the stated question (namely, is the research hypothesis true?).

Already, we are touching on the foundation of statistics. We just talked loosely about “the probability that H is true,” but one must then also be willing to assign such a probability to H. The methodology we discussed was developed when this was a controversial idea, to say the least. The prevailing philosophy was that H is simply either correct or incorrect, and that probability has nothing at all to do with it. The researchers must determine the actual reality, which is unknown but unshakably fixed. On the other hand, an experiment’s outcome is definitely subject to the twists and turns of fate: a better medicine can, by chance, cure more or fewer persons in a given group of patients than expected on average. So one can speak of the probability that an experiment has a particular outcome if the null hypothesis or research hypothesis is true, but not of that which interests one more: the probability that the null hypothesis or research hypothesis is true if the experiment has a particular outcome.

At the same time, the impossibility of speaking about what one really wants to know is also why incorrect interpretations are difficult to avoid. What one wants to know is whether the research hypothesis is true based on the data. However, the significance and power of the experiment do not say anything about this: they predict the probability of a statistically significant outcome assuming that the research hypothesis is true or not.

We have now discovered that the classical setup for testing hypotheses cannot lead to a satisfying and straightforward procedure. Indeed, depending on the field, the chosen significance level, and the power of the procedure, it may even happen that (by far) most claims in the field based on significant outcomes are false.

Of course, the reality of statistical testing will not be as simple as Ioannidis describes it here. Statistical analyses are certainly not always purely aimed at obtaining significant outcomes to draw immediate conclusions from. But Ioannidis’ analysis does show what can go wrong when such testing is blindly applied on a large scale. So why is such testing used in the first place? To understand this, we go back to the most important founder of significance testing.

1.2The Logic of Significance Testing: In the Words of Fisher

The procedure outlined in the previous section has its origins in the work of Ronald Fisher, one of the founders of modern statistics. We will first let him speak himself.

In his 1935 book The Design of Experiments [10], Fisher introduces the problem with a beautiful, typically British, example. Suppose, he says, that a lady claims that when she is served a cup of tea with milk, she can taste whether the tea or the milk was poured first. How could we test whether such a claim is true?¹ An experiment to verify this should of course consist of repeatedly offering the lady both variants and keeping track of how often her judgment is correct. But how often must it be correct for us to believe her? This will inevitably be partly arbitrary, because no matter how many times she is right, it always remains possible that she is extraordinarily lucky and has guessed everything correctly. All an observer can do is, as the word suggests, observe. Deciding whether the lady’s claim is correct, however, requires more than that: perhaps one observer is more likely to think that making this distinction is possible and will therefore change their mind and decide that the lady’s claim is correct at a smaller number of correct classifications than a more skeptical observer.

Fisher proposed a protocol in which one first agreed on how much chance to accept at most as an explanation, before deciding to believe the lady. In this example, he chooses 1 in 20, or 5%. Let us see what type of experiment would then be needed.

To give the lady the opportunity to show her ability to distinguish between the two variants (tea or milk poured first), we give her a series of cups of tea in which both occur. We can, for example, prepare six cups, three of each kind, inform the lady of the setup, and ask her to point out which cups are of which kind. What if she gets them all right? How difficult is that? We now need to know the probability that the lady would have guessed right if her claim were false.

We can calculate this if we know the experimental setup and how often the lady would guess correctly if her claim were false. We therefore, says Fisher, look at “a subject without any faculty of discrimination,” i.e., someone who guesses and cannot make even the slightest distinction. As we know that there are two groups of three cups, we must look at the number of ways to divide six cups into two groups of three. The lady can get the correct answer by guessing correctly if she points out the three cups in which milk was poured first purely by chance; the other three are then automatically also correct. There are many ways to choose a group of three cups from a total of six. For the first cup, we have six to choose from; for the second, we still have five, and for the third, four choices are left. This gives a total of 6 × 5 × 4 = 120 ways, but every combination of three is obtained six times: for example, the cups 2, 3, and 5 can be chosen in six orders (2, 3, 5), (2, 5, 3), (3, 2, 5), (3, 5, 2), (5, 2, 3), and (5, 3, 2). So in all, we have only 120/6 = 20 distinct choices for the three cups in which we first poured milk: amateurs therefore have a 5% probability of bluffing their way through the test.

This is (coincidentally or not) exactly the percentage that Fisher considers maximally acceptable for this kind of testing, so this experiment may or may not suffice. But if we expand the experiment (we will do so shortly), it naturally becomes increasingly difficult to keep guessing correctly by chance. Using the same reasoning as before for a total of six cups, we see that there are 70 ways of choosing four out of eight cups into which we first pour the milk. A total amateur then only has probability 1/70 of guessing everything correctly by chance.

If the lady gets everything right (with eight cups), Fisher says that the amount of chance required to pull this off when she has no distinguishing ability is now too great (in other words, the probability is too small) to keep believing that her claim is not true. We have already found the first, most important, requirements of an experimental setup as envisioned by Fisher, which we list below:

–In the experimental setup, we first assume that the claimed effect does not exist: we call this the null hypothesis.

–We design an experiment that can produce an outcome that is very unlikely if the claimed effect does not exist, that is, if the null hypothesis is true.

–If we do get such a result, we no longer believe that the claimed effect does not exist, and we decide that it does exist; that is, we reject the null hypothesis.

Note that this list does not describe what we do if the experiment yields something that is not part of the outcomes that lead us to reject the null hypothesis. On this subject, Fisher says:

[…] it should be noted that the null hypothesis is never proved or established, but is possibly disproved, in the course of experimentation. Every experiment may be said to exist only in order to give the facts a chance of disproving the null hypothesis.