Few students sitting in their introductory statistics class learn that they are being taught the product of a misguided effort to combine two methods of inference into one. Few students learn that many think the method they are being taught should be banned. In Understanding Null Hypothesis Tests, and Their Wise Use, I emphasize Ronald Fisher’s approach to null hypothesis testing. Fisher’s method is simple, intuitive, thoughtful, and pure. If we follow Fisher’s example, all the criticisms of null hypothesis tests melt away. Fisher on a good day. Do you collect data and then ask a friend how to analyze them? Once you have read this monograph, you will not have to do that anymore (at least not often). You will understand the concepts behind the mathematics. You will see why different types of data require different types of tests. Do you think that P is the probability of a type I error? It is not. If you fail to reject, do you accept the alternate or alternative hypothesis? You accomplish nothing by doing so. Do you equate statistical “significance” with importance? You should not. These and other misconceptions are explained and dispensed with in Understanding Null Hypothesis Tests, and Their Wise Use. After reading this monograph, you will understand why it is utter foolishness to say, We use confidence intervals instead. (Confidence intervals are wonderful, but they show the results of null hypothesis tests performed backwards.) More importantly, you will understand wise use. You will use P-values thoughtfully, not to make mindless, binary decisions. Most importantly, you will know the Big Secret that should not be a secret. A null hypothesis is infinitely precise, so many and maybe most null hypotheses cannot be correct—a fact we should know from the start. It was the legendary statistician John Tukey who explained why it is important to test such nulls anyway: to determine whether we can trust our data to tell us the direction of a difference. Getting the direction wrong is referred to as a type III or type S error. Once you have read about type S errors in Understanding Null Hypothesis Tests, and Their Wise Use, you will have a better understanding of null hypothesis tests than anyone on your block.
Corotto, Frank, "Understanding Null Hypothesis Tests, and Their Wise Use" (2020). Faculty Publications. 2.