What is MDE? This article is one in long line of series on AB Testing that we intend to post. I expect that the reader has some familiarity with AB Testing and article intends to be a revision capsule for the reader.
What is MDE?
Minimum detectable Effect is minimum increase that is expected out of implementing the A/B Test.
The measure is an effective means to identify how much your visitor sample size should you estimate before testing.
The visitor size is integral to the cost of testing, more the visitors or the sample size , more is the time required to achieve that much of critical mass.
The risk associated with this measure is the “hard to predict” nature of the measure.
Effect is the quantitative measure of strength of the change in phenomenon. Or the percentage difference between the null measure of hypothesis and the alternate measure of hypothesis
ffect is the quantitative measure of strength of the change in phenomenon. Or the percentage difference between the null measure of hypothesis and the alternate measure of hypothesis
How does one account for it?
One can’t predict “actual” increase or decrease that will happen on account of change that is being introduced on the page. But one does have the following details:
Current daily visitor count on the page
Current conversion rate
Acceptable level of statistical significance (95% generally)
Some expected minimal increase in the conversion rate
We can add these sample size calculators like ones below to fairly estimate that sample size that is required to achieve expected minimal increase in conversion rate.
https://www.optimizely.com/sample-size-calculator/?conversion=10&effect=10&significance=95
Assuming the sample size requirements is 13K visitors, we make the following considerations:
Based on our daily visitor count level, in how many days or weeks above level of sample will be achieved.
Can business afford the time taken for MDE to be achieved?
Acceptable level of best practice is to have a range of MDE, if your baseline conversion rate is 30%, and you set an MDE of 10%, your test would detect any changes that move your conversion rate outside the absolute range of 17% to 33%.
Why is it important?
MDE helps you decide an accurate trade-off between the sensitivity of the change vs how long would it take to measure significant impact of the change. Higher the time you take to assure the achievement of MDE will directly impact the sample size or the time associated with the A/B Test. More the time , more is the cost associated with the AB Test. It is also a trade off against any other experiment that one might be trying to run on the same page or website.
Summary
The smaller your MDE is, the larger the sample size required to reach statistical significance.
One should carefully assume it considering time and cost allocated with the test(vis-à-vis some other test which can be give better results in same time)
References?
Eddie Jaques
+61413785621