We are often asked how much data is enough to make decisions during conversion rate tuning.
[WARNING to marketing people: basic math ahead!]
Let’s look at the simplest example: you are trying to decide whether option “A” or “B” is best. You have split your traffic equally to test both options and have gotten 90 conversions on A, and 100 conversions on B.
There are two questions that you can ask:
1) “Is B really better than A?”
and
2) “Assuming that B really is better, how much better is B than A?”
Even though it might be of interest to you, the answer to #2 requires A LOT more data than the answer to #1. Since you should be continuously optimizing all aspects of your marketing program, you need to make good decisions quickly and move on.
So let’s focus on question #1. You just want to know WHICH ONE is better – not by HOW MUCH. As soon as you know which version is better, you should “flip the switch” and start using that one on all of your traffic because it will make you more money.
With statistics you can never be absolutely sure of the answer, but you can get a very high “degree of confidence” if you collect enough data. The amount of data required depends on the confidence level that you want. Do you need to be right 3 out of 4 times, 9 out of 10, or 99 out of 100? Once you have picked your confidence level, you simply wait to collect enough data to reach it.
This confidence can be expressed by means of a “Z-Score” which is very easy to calculate. Z=1 means that you are 67% sure of your answer, Z=2 means 95.28% sure, and Z=3 means 99.74% sure.
Lets pick a 95% confidence level for our example above. This means that we want to be right 19 out of 20 times. So we will need to collect enough data to get a Z-score of 2 or more.
The calculation of the Z-score is related to the Standard Deviation (“SD”) in statistics. A Z-score of 1 is the same as one standard deviation from our current value. A Z-score of two is the same as two standard deviations from our current value.
SD= square_root(Conversions) / Conversions
in our example for B
SD = square_root (100)/100 = 10/100 = 10%
So we are 67% sure (Z=1) that the real value of B is between 90 and 110 (100 plus or minus 10%). In other words, there is a one out of three chance that A is actually bigger than the lower end of the estimated range above, and we are just seeing a “lucky streak” for B.
Similarly at our current data amounts we are 95% sure (Z=2) that the real value of B is between 80 and 120 (100 plus or minus 20%). So there is a good chance that the 90 conversions on A are actually better than the bottom end estimate of 80 for B.
If we wanted to be 95% sure that B is better than A we would need to collect much more data. In our current example this level of confidence would be reached when A had 1350 conversions and B had 1500 conversions. Note that even though the ratio between A & B still remains the same, the Standard Deviations have gotten much smaller.
This may all seem a little intimidating at first, but the math for these calculations can easily be programmed into an Excel spreadsheet formula. After that, you just plug in the numbers and see if your desired confidence level has been reached yet.
Believe me, this is preferable to making the wrong decision one third of the time like in the example above.
Posted by sitetuners 
Posted by sitetuners 
Posted by sitetuners 
