All Collections
A/B Testing & Optimization FAQ
Why do you show improvements in form of intervals?
Why do you show improvements in form of intervals?
Lari Lehtonen avatar
Written by Lari Lehtonen
Updated over a week ago

It would be wrong not to do it as such! Or in other words, it would be a risky shortcut.

As with all tests, reaching a significance test result relies a lot on the volume of data gathered and the conditions met during the test period. The pool of data available in a given test (also known as sample size) leads to calculate the probability that a variation in a test is winning. As with each distribution, the data can be dissected to look at various intervals, using different percentile or, in a more simplistic manner, using the median value of the distribution for example. The median is the value that splits the data set in two, such that 50% of the data points are smaller and 50% are larger than that value. As more and more data are aggregated by Nosto, the interval usually tends to get narrower. 

The improvement interval shown by Nosto represents the range of values where we are 95% certain that the true improvement will be contained in. In other words, if test would be run again with the exact same condition met, the improvement would fall into the same interval. On hover, you have access to additional data points such as the middle 50% range of the data that are most probable and the median value. 

Nosto uses advanced statistical methodologies that take into account various factors within a test, such as time, context and data consistency. Simplifying the maths and dividing one number by another can't account for all of these factors. The ultimate goal is to best forecast how the variations are likely to perform in the future and to avoid so your results are more likely to remain valid in the long run after a test is ended.

Did this answer your question?