Significance, Importance and Equality—Three Basic Concepts in the Analysis of a Difference

Abstract

By means of data from ficticious cross over trials, it is first demonstrated that a statistically significant difference is not necessarily of a practically important order of magnitude. This fact is of special interest when the number of observations is large. Second, a statistically non significant difference does not prove the hypothesis about equality between, say, treatment effects. This fact is of special interest when the number of observations is small. For investigating whether equality is plausible, confidence intervals are more useful than non significant results from tests of significance.