Many reasons can affect why a test is not significant. Some of these reasons can for example include:
- There is not enough traffic in the targeted segment.
- Some variations perform very similarly.
- The test has been live for less than 14 days (Nosto declares significance at earliest day #14 in the reporting view).
- The base variation is already optimized.
- The changes between variations are too small. For example, one might want to maximize the total number of baskets created by users visiting product pages. Copy change of the Add-to-cart button can be a way to make it clearer for users to reflect the purpose of the button to be aligned with the user’s intent. However, if the existing copy being “Add To Bag” is already clear, challenging it with a “Add Product To Bag” might not likely be perceived as a clear improvement for users to best understand the purpose of the button. Color changes or modifying position of the button on the product page will most likely yield more drastic improvements.