All Collections
A/B Testing & Optimization FAQ
What if my test is not significant?
What if my test is not significant?
Lari Lehtonen avatar
Written by Lari Lehtonen
Updated over a week ago

There are various options, all depending on the hypothesis stated and the overall purpose of the test. However, some options are always available regardless of those.
​ 

  • Extend the reach. If your segment is too narrow, try to target a segment reaching more users. 

  • Remove some variations. If you have for example created 4 variations, try to run the same test with 2 or 3 variations. 

  • Adjust settings or design of campaigns within variations. A good starting point is to make - if possible - the variations more distinctive from one another as it currently was in the initial test.

  • Change your focus (Optimization Goal).

  • Keep it running!

  • Follow your instinct and pick what is best aligned with your brand. At similar results between two variations, if one variation is agreed by all departments that it’s more aligned with your brand guidelines than the other, you may want to select it as a winner. 

  • Try running the test further up the page or further down the page! 

  • Run the test again! It’s good practice to run the same test twice to validate or invalidate the initial results. Conditions will most likely not be the same (different time period, traffic fluctuation, …) so results might be different!  

  • Let it be. If you haven’t managed to challenge the status quo, it can be that your base variation is already optimized. Time to focus to another area of optimization. 

  • Analyze further - Use segment filtering to look at the results from any segment perspective to identify opportunities. 

  • Analyze further - Use Merchandizing Insights to better understand why a variation underperform or outperform during a test.

Note that Nosto informs users if there is an estimated time to wait to reach significance in the Analysis panel:

You can learn more about the different states of Nosto tests here.

Did this answer your question?