A lot of factors influence the results of any A/B test. From the intrinsic performance of each variation that are part of a test or the stability of each of those variations over time for example, it can be challenging to interpret the results of a test and to decide which variation should remain live at the end of a test. At Nosto, we have built a set of functionality, making it easy to understand in which state a test is in a given time, how variations are performing in a test and why they are performing as such. 

In this article, we will cover how to interpret the results of Nosto A/B tests using the reporting view, based on three typical use cases, positive, negative and neutral and other related details

In any Nosto test, the performances of some variations are compared to the performance of a base variation, serving as a control or baseline variation. It’s important to remember that any variation can be chosen to be a base variation at setup time using the Nosto testing wizard.

For the sake of simplicity and consistency, we will illustrate how to interpret results focusing on Conversion Rate optimized tests where two variations are tested against each other. However, the same analysis methodology applies to tests where more than two variations are tested against each other and if other optimization goals would be selected. 

First, let's explore the results of a significant test where a variation is winning over the base variation and yields a positive improvement on Conversion Rate, the optimization goal selected in this particular use case.

In this test, there is 98.63% probability that the Variation B is winning over Variation A (Base variation) on Conversion Rate. Selecting the variation B, can yield an improvement compared to the base variation between +0.49% and +7.85%, indicated by the boxplot.
​
If you hover over the boxplot the full range of expected improvements can be explored, looking at the lower and upper bound values and the expected median improvement: 

In this example, it's safe to say that the expected improvement is around +4.08% but there is an equal probability that it falls between (in the most pessimistic scenario) +0.49% and +4.08% or between +4.08% and +7.85% (in the most optimistic scenario). In either case, the improvement over the variation A (Base variation) would still be positive, which is a very positive signal for choosing this variation!

Another way to interpret the results is that if the same test would be run 10,000 times in the exact same condition, the Variation B would still win over the Variation A about 9,863 times with an improvement of the Conversion Rate that would most likely fall between +0.49% and +7.85%. 

If for any given reason, the Variation A would be selected, the potential loss of relative Conversion Rate would be 7.23%, which can be interpreted as the cost of making the wrong decision. In any test, the lower the regret (the loss of conversion opportunity) is for a variation, the better. However, regret can also go above 100%, depending on performances of each variation in a test. 

Note: An alternative interpretation of regret is by How much better may (in the worst case) another variation perform based on the observed performance in a given test.

Let's now take a close look at the results of a significant test where the Variation B doesn't yield a positive improvement but rather negatively impact the Conversion Rate of the Variation A (Base variation). 

In this test, there is 100% probability that the Variation A (Base variation) is winning over Variation B on Conversion Rate. With the Variation B, can be expected a negative impact on the Conversion Rate of the base variation between -2.21% and -3.64%.

If the boxplot is hovered, the full range of improvements can be explored, looking at the lower and upper bound values and the expected median improvement: 

In this example, it's safe to say that the expected improvement (or, negative impact in this case!) is around -2.94% but there is an equal probability that it falls between (in the most pessimistic scenario) -3.64% and -2.94% or between -2.94% and -2.21% (in the most optimistic scenario). Either case, the improvement over the variation A (Base variation) would not be positive, which is a very strong signal for not choosing this variation and enabling the base variation instead.  

If for any given reason, the Variation B would be selected, the potential loss of relative Conversion Rate would be 3.65%, which can be interpreted as the cost of making the wrong decision. In any test, the lower the regret (the loss of conversion opportunity) is for a variation, the better.
​
​

Finally, the third scenario is focussed on a non-significant test. In the test below, none of the variations has a high enough probability of winning, or in other words, above 95%. 

In this test, there is only 81.99% probability that the Variation B is winning over Variation A (Base variation). With the Variation B, can be expected an improvement on the Conversion Rate of the base variation between -0.24% and +0.66%. In other words, the improvement can either be positive or negative. 

If the boxplot is hovered, the full range of improvements can be explored, looking at the lower and upper bound values and the expected median improvement. 

In this example, it's safe to say that the expected improvement is around +0.21% but there is an equal probability that it falls between (in the most pessimistic scenario) -0.24% and +0.21% or between +0.21% and +0.66% (in the most optimistic scenario).

It can be challenging to know if it's the right choice to choose Variation B. However, in this case, if it would be selected, the potential loss of relative Conversion Rate would only be 0.17%. In this case, it's important to also notice that the Variation A also has a regret rate below 1%, at 0.58% exactly. If Variation A would be selected, the potential loss would also be very minimal.

You can read more about the next steps when a test is not significant in this article.
​

In general, relying purely on data in the world of A/B Testing is a safe bet but it only tells half of the story as it never explain why variations are winning or losing. Beyond intrinsic results, there can be a myriad of reasons why customers would behave in a certain way and shop more when they are exposed to a given variation. 

For example, would one still confidently declare that a variation is a losing variation if one would know that a majority of the products users ended up viewing were temporarily discontinued because of a deficiency in your buying process or some inventory issues? 

It's always recommended to analyze further to really get to the bottom of the results by backing up number based test analysis. Nosto's layer of Merchandizing Insights helps to unveil and identify these opportunities. You can learn more about Merchandizing Insights here.  

If you want to learn more about how the attribution works for any Nosto A/B test, you can find a dedicated article here.