A/B Testing, calculating lift rate of a test
One of the determinant factors of two-tailed hypothesis testing is the lift rate. What then is lift rate, its importance in A/B testing and how is it calculated? Let’s begin………
As you would have already known by now, A/B testing involves two groups namely control_group and treatment_group. The whole essence is to determine if the new improvement would lead to a significant result compared to the initial. One of the key determinants is the Lift rate.
Lift rate is calculated to tell by how much has the treatment_group changed to the control_group. This can be a positive or negative effect. Do not always expect a positive result as you are calculating this to determine the impact which could go both ways. What is the importance of calculating the lift rate of A/B testing? The most important reason for this is to determine if the would be a significant change in the output in the event the new feature is fully rolled out to all users. If there is a negative output, then it would be unproductive and rollout the opposite is true. Let’s take a practical example.
Practical example
As an e-commerce company, you plan on giving all inactive members $6 when they reactivate after sending them a reactivation email message. This was rolled-out and you had the following details. Total controlled_group was 391 which resulted in 100 being converted and reactivating whiles treatment_group was 395 which also resulted in 170 conversions. Calculate to find out if the treatment_group performed better than the controlled_group and if it’s prudent to rollout the new strategy broadly.
The First step is to calculate the conversion rate for the various groups. What then is the formula for calculating that?
So let’s calculate the conversion rate for the controlled_group from the question above.
Let’s also calculate the conversion rate for the treatment_group from the question.
With the various calculations let’s calculate the lift rate and interpret the result.
Interpretation of result
This indicates that the treatment group has shown a higher level of impact than the control. This means the treatment has performed by 68% than the control group. In other words, the alternative hypothesis is more impactful than the null hypothesis. And based on this, the feature should be rolled out into full production.
But to have a more valid impression and confidence about this finding, there is a need to compute and find the statistical significance of the results.
I hope this have been helpful to help you as a product analyst or data analyst. In my next article we would look at how to calculate this using SQL which would come with some advanced checks on the data. A/B testing is a very important skill as an analyst you need to help product managers make informed decisions based on feature rollout. Thank you for reading. Contact me via email on henrykpano@gmail.com with your questions. Share and follow me.
