A/B Testing For Practical Significance

When doing statistical hypothesis testing, the math behind it gives us the toolset to determine the statistical significance of our observations. But if we’re doing a two-sample test on a simple hypothesis, eg. $H_{0}:\mu_{1}=\mu_{2}$ vs. $H_{1}:\mu_{1}\ne\mu_{2}$ rejecting it won’t tell us anything about the magnitude of the difference. Usually aside from making sure that the difference in measurements you observed are statistically significant, you want your observed differences to be practically significant as well. That…

Continue reading