Nosto's A/B Testing & Optimization allows you to understand how different experience variations across your site affect the behavior of each individual visitor. With our Continuous Optimization tool, you'll never have to worry about turning customers away with unfavorable onsite experiences.

In this article, we uncover the key principles of Nosto's A/B Testing & Optimization product to help you take full advantage of our tool, along with walking through its different views and features.
​

A/B Testing & Optimization Wizard

The A/B Testing & Optimization Wizard is a 3-step guide to help you create and configure any Nosto test, allowing you to easily follow our 4 W's testing framework. 

In the first step of the wizard, you can choose between different test types and methods, as well as name and describe tests.

Once a test name is given, it can be saved as draft. To move on to the next step of the wizard,  you can click Next.

The second step of the wizard allows you to select which metric you primarily want to optimize for. This essentially helps you create different variations and get a preview of how these variations would create different experiences on your site. This step also helps you decide whether or not you want to continuously optimize for best performance during a test.
​
Optimization Goals are bucketed into groups: 

Once an Optimization Goal is selected, it's time to create variations. Let's take a closer, step-by-step look at the variation creation phase.
​

Segment Selection - Who?

First, using a dropdown menu, you can select which segment of customers will be included in a test. You can select any Nosto segment, lifecycle automated segments or custom segments, but as a rule of thumb, it is best to start by using largest audience possible, everyone, to ensure the test has enough traffic. Naturally, if the experience subjected to test is for a specific segment only, reflect this in the selection. It's also possible to target multiple segments at the same time, and we recommend you to merge these segments using our segmentation capabilities. Once a segment is selected, it's time to focus on creating variations using Nosto placements and Nosto campaigns.
​

Placement selection - Where?

To create variations, you need to define where you want your test to run on your online store. As campaigns are displayed into Nosto placements, you first need to select which placement(s) you want to use as part of a test. A Placement selection modal helps you identify a placement:

​Campaign selection - What?
​
Once placement(s) are selected, it's time to build variations and select which campaign(s) should become a variation A, a variation B and so forth. Campaigns are the actual personalisation features, content, recommendations or UGC. Note that the variation A is always the base variation in any test. All performance of all variations are always reported against the base variation in a test. 

When selecting a campaign, a campaign selection modal helps you identify which campaign should be set, per placement and per variation:

Above is an example of campaign test, where three content campaigns will be A/B tested on the front page in the 1. Hero Banner placement.
​
​
​Previewing variations 

When you are satisfied with the variations created, it's possible to preview how experiences look like on your site using the previewing tool. When clicking on Preview 3 variations in the upper right corner, a modal displays a summary of the variations along with specific redirection preview links:

Once this is done, users are redirected onsite, with the Nosto Debug Toolbar now on. 

Continuous Optimization 

Once variations are set, you can choose how the traffic is split during the test and automatically drive site traffic to your highest performing variations—avoiding poor performance and empowering constant optimization.

If disabled, you can adjust the split manually to match 100% of visits within the targeted segment.

If you want to learn more about our Merchandising Insights, check out this blog post which dives deeper into the value of this feature.
​

Summary
​
A Summary step enables you to save and activate a test:

Note that if some campaigns and/or placements used in the settings are currently disabled, you will be informed by an extra modal allowing you to enable all campaigns and placements concerned.
​

Test Listing View
​
Once created, all tests are listed in the test listing view. This allows you to have an at-a-glance view of key information about each test: such as names, test methods, date of creation, how long the test has been running live, the optimization goal selected, the state of any given test and its status.

Tests on draft can be discarded using the button on the bottom left corner of the wizard:

Live tests can be ended using the Actions dropdown menu on the top right of the page:

In a similar manner, tests that have been ended can be archived on the reporting view using the Actions dropdown menu on the top right of the page:

You can learn more about all states of Nosto tests here.
​

Reporting View
​
When tests are live, ended or archived, you can explore the results and the performance of each variation in a test on the reporting view:

First on the list is the segment of customers targeted by a given test, allowing you to filter and dissect the results from any segment perspective. The test name, along with the list of actions that can be accomplished, are listed in the Actions dropdown menu on the top right corner of the page.

A high level summary is available on the top of the page that displays how long the test has been live, the probability that one variation is winning, the total number of visits aggregated by all variation in a test, the optimization goal, the state of the test and a dedicated analysis panel. You can find more information about all Nosto test states here.

The line graph panel informs you about the performance of all variations on a daily basis, both from an optimization goal and traffic allocation perspective. Note that traffic allocation differs potentially on a daily basis when continuously optimized. 

Here's an example of trend when one variation (B) has clearly won against another variation (A): 

Below the line graph exposing the trend, a variation table enables you to deep-dive into the performance of each variation—both from a primary optimization goal and secondary metrics perspective. 

You can learn more about how to interpret results here.
​
​Merchandising Insights
​
Merchandising Insights helps you surface deeper-level information on how your personalization and optimization strategies affect sales for individual brands, products or audience segments. 

It's possible to explore which products, brands and categories have been viewed and purchased by users bucketed in those respective variations. 

If you want to dive deeper into the benefits of using our Merchandising Insights feature, check out our blog post.
​