Nosto's A/B testing capabilities for Search & Category Merchandising allow you to understand how different merchandising rule variations are performing for certain queries, on specific category pages and for chosen segments. By understanding the performance of your variations you can make sure to always deploy the best performing merchandising rules across your shop.

In this article, we have a close look on how to get started with this tool, walking you through the steps to create your first A/B test for merchandising rules.

The A/B Testing & Optimization Wizard is a 3-step guide to help you create and configure any Nosto test. In the first step of the wizard, you can choose between different test types and methods, as well as name and describe tests.

If you want to test merchandising rules targeting Categories (either all categories or specific category pages), choose Category Merchandising as test type.

If you want to test merchandising rules targeting Search (either all queries or specific search queries), choose Query Merchandising as test type.

Once a test name is given, it can be saved as draft. To move on to the next step of the wizard, you can click Next.

The second step of the wizard allows you to select which metric you primarily want to optimize for. This is the essential goal of the A/B test to understand how different variations are performing related to the specified metric.

Your next step is the definition of the variations that you want to include in the A/B test scenario. Click on Add variation to start adding merchandising rules that you want to compare. You can find a list of all available merchandising rules in the modal that opens when you click on the pencil.

Please note that we recommend to only compare merchandising rules that have the same target and segment - this way you can guarantee a statistically meaningful outcome.

If you want to edit your rules before adding them as a variation, you can directly jump into the rules menu from the modal by clicking on Manage merchandising rules.

As a last configuration on this page, you can adjust the traffic split between the different variations.

Please note that Continous Optimization is not available for testing merchandising rules on Search and Category Merchandising.

By clicking Next, you get to the last page in the wizard. Here you can find a summary of the A/B test that you are going to launch. Please make sure to check if you've set up everything correctly. You can now either save the draft to come back to this later, click previous to adjust the details or click Save & Activate to start the A/B test.

Once created, all tests are listed in the test listing view. This allows you to have an at-a-glance view of key information about each test: such as names, test methods, date of creation, how long the test has been running live, the optimization goal selected, the state of any given test and its status.

Live tests can be ended using the Actions dropdown menu on the top right of the page.

In a similar manner, tests that have been ended can be archived on the reporting view using the Actions dropdown menu on the top right of the page:

You can learn more about all states of Nosto tests here.

Good to know: You can also preview running A/B tests by using the preview tool if you want to get a snapshot at a certain time.

When tests are live, ended or archived, you can explore the results and the performance of each variation in a test on the reporting view:

First on the list is the segment of customers targeted by a given test, allowing you to filter and dissect the results from any segment perspective. The test name, along with the list of actions that can be accomplished, are listed in the Actions dropdown menu on the top right corner of the page.

A high level summary is available on the top of the page that displays how long the test has been live, the probability that one variation is winning, the total number of visits aggregated by all variation in a test, the optimization goal, the state of the test and a dedicated analysis panel. You can find more information about all Nosto test states here.

The line graph panel informs you about the performance of all variations on a daily basis.

You can learn more about how to interpret results here.

Merchandising Insights helps you surface deeper-level information on how your optimization strategies affect sales for different audience segments.