Document Type

Article

Publication Date

Winter 2-7-2020

Abstract

Many interpret error bars to mean that if they do not overlap the difference is statistically significant. This overlap rule is really an overlap myth; the rule does not hold true for any conventional type of error bar. There are rules of thumb for estimating P values, but it would be better to show error bars for which the overlap rule holds true. Here I explain how to calculate what I call comparative confidence intervals which, when plotted as error bars, let us judge significance based on overlap or separation. Others have published on these intervals (the mathematical basis goes back to John Tukey) but here I advertise comparative confidence intervals in the hope that more people use them. Judging statistical significance by eye would be most useful when making multiple comparisons, so I show how comparative confidence intervals can be used to illustrate the results of Tukey tests or Dunnett’s test. I also explain how to use of comparative confidence intervals to explore the effects of multiple independent variables and explore the problems posed by heterogeneity of variance and repeated measures. When families of comparative confidence intervals are plotted around means, I show how box-and-whiskers plots make it easy to judge which intervals overlap with which. Comparative confidence intervals have the potential to be used in a wide variety of circumstances, so I describe an easy way to confirm the intervals’ validity. When sample means are being compared to each other, they should be plotted with error bars that indicate comparative confidence intervals, either along with or instead of conventional error bars. This paper is based on a submission that was rejected by Psychological Methods. The original submission along with the reviewer’s comments and my responses are available at the bottom of this page.

Comments

I am thankful for Enes Aganovic’s help in preparing the figures and Shane Webb’s feedback on a draft of this article.

Submission to Psychological Methods.docx (652 kB)
The submisson this publication is based on.

Reviewer's comments and my responses.docx (23 kB)
A reviewer's comments and my responses

Critical values for 0.15 CCIs.docx (33 kB)
Critical values of q needed for 0.15 comparative confidence intervals

Share

COinS