The Fuss with Statistical Significance Testing

One of the basic skills a web analyst should know is how to perform and interpret significance testing. Many look at web metrics –5,071 daily unique visitors, 102 orders, 80% fallout rate, 5% checkout conversion, etc. – and see nothing but plain numbers. Can you say a 500 sales count is notably higher than a 515 sales? Is 6% email open rate comparative to 9%? Is the decline in unique visitors as drastic as it looks? If these are the types of questions you have in mind, then you ought to be aware that you should at least be doing statistical significance testing.

Statistical significance, since it is based on a computation with some elaborate probability and confidence level, gives a more scientific basis to claim if two or more means (or measures of central tendency; proportions/rates; data distributions; and independence of data among k groups) are considerably different. Knowing the significant difference between two mean or more means gives us sufficient evidence to decide that one value is much higher or lower than the other and if they are, it would imply another meaning to just declaring “We hit the target sales by 30!” or something like that.

Before delving into your actual testing, you need to understand the requisites in doing significance testing.

• Null Hypothesis – Every test of significance starts with this. It is a speculation usually believed to be true but has not yet been proven. By the way, in a statistical point of view, the null hypothesis is commonly stated as an equality so for an instance, my null hypothesis could be that “Changing the color of the call to action button, from the default blue, to green does not increase customers click through it”
• Alternative Hypothesis – This is the statement you want to result to. An alternative to the statement above may be “Click through on green call to action button is higher than the blue one.”
• α, the Level of Significance – The criterion used for rejecting the null hypothesis
• Test Statistic – A random variable which value serves to determine the conclusion
• Critical Value – A tabular value that the test statistic needs to exceed in order to reject the null hypothesis. This goes hand in hand with α.

Now that these terms have been introduced, following the steps in statistical testing would be easier. Comparing the status of the site’s key metrics through their significant difference over the past few months will give more meaning to data. This will also give Web Analysts more confidence when reporting the real impact of the increase and decrease of the site’s key metrics.