Excerpt from free eBook, "The Essential Guide to A/B Testing for Digital Advertisers"

Marketers perpetually ask if test results are “statistically significant,” but what does that term even mean?  Probably not what you think.  

Traditional statistics relies on p-values as a measure of the “statistical significance” of test results.  Most marketers are surprised to learn that p-values do not, in fact, measure the probability that one creative or treatment will outperform another.  What p-values measure is far more abstract and removed from the decisions that marketers make based on A/B tests.

In every A/B test one variation will perform at least slightly better than the other.  P-values measure the probability that a test result (say, creative variation B outperforms creative variation A by 10%) would have occurred if in fact there were no difference between the two creatives at all.  That “95% confidence level” threshold you’ve probably heard bantered about simply means that there is a 5% chance that, were the two variations identical, you would have observed as large a difference in performance between them as you did.  The p-value is an important measure in other fields of study to account for what is known as in traditional statistics as Type I error.4In our experience, we have yet to hear a digital marketer ask us for this specific probability.  And why would they.

Learn why Adacus uses Bayesian statistics in our latest eBook

"The Essential Guide to A/B Testing for Digital Advertisers"