A/B testing with Flows

Recharge's Flows is designed to simply and intuitively build comprehensive customer journeys. A/B testing is a core functionality of all Flows, including pre-built templates. Flows enables you to run A/B tests on different incentives and offers to determine which marketing efforts actually increase retention and AOV throughout the customer journey.

This guide covers benefits of A/B testing and how you can run an A/B test in Flows. 

A/B testing overview 

A/B testing, also known as split testing, is a marketing research methodology that compares two versions of a marketing campaign to determine which one performs better. This is done by randomly dividing the target audience into two groups, and then showing each group a different version of the campaign. The results of the test are then analyzed to determine which version of the campaign performs better.

A/B testing makes it easy and accessible for anyone to build a hypothesis for a new customer experience and to test different iterations of that with your customers to optimize your engagement & revenue metrics.

Benefits of A/B testing

The following lists some benefits of using A/B testing:

• Increased engagement: A/B testing can help you identify the elements of your store that are most engaging to customers, so you can focus your efforts on those areas.
• Improved conversion rates: By testing different versions of your store, you can identify the changes that lead to more next order conversions.
• Increased revenue: A/B testing can help you increase your revenue by optimizing your growth & retention strategies specific to next order conversions and average order value (AOV)

A/B testing with Flows

Step 1 - Determine the goals of your test

1. What: hypothesis would you like to test?
   • Determine if offering a free gift on a specific order sequence actually improves retention. Set up a Surprise and Delight flow to test this hypothesis. 
   •  Identify which upsell offers increase subscriber AOV and LTV. Set up a Targeted Upsell flow to test this hypothesis. 
2. When: What part of the customer journey would you like to test? When do you want the flow to trigger? Common questions to ask when determining trigger order number:
   • Is there a specific charge where most of your customers cancel? I.e. after 2, 3, 4 orders? Would incentivizing customers with a free gift after this order help improve retention? 
   • Do you want to expose all loyal customers to a new product and allow them to add to their next subscription at a discounted price?
3. Who: Which customers do you want to target? Gain deeper insights by targeting users based on their characteristics, such as their recurring order number, AOV, LTV, location, or product usage.
   1. Order filters: Target customers who ordered or didn’t order a specific product(s).
      • Do certain products complement each other? Add a free gift or targeted upsell offer to customers who already purchased one of those products.
      • Avoid sending customers duplicate products by excluding customers who subscribe to the incentive product.
   2. Customer filters
      • Target your most loyal customers and limit the incentives to customers with a higher-than-average AOV or LTV.
      • Target customers based on location.
4. Time: How long do I want to run the experience and how many customers do I want to participate?
   • Do you only want the test to run for a limited time? Set a start and end date.
   • Do you want to limit the number of customers that can enter the flow? Do you have a limited number of free gifts to send out? Set a customer limit.

Once you have answers to these questions, you can configure the flow in Recharge.

Step 2 - Create an A/B test

To create an A/B test in flows:

1. Click Flows in your merchant portal. 
2. Click the Create New dropdown and choose AB Test Flow. Optionally update the name of the Flow by clicking the pencil icon next to the title. Select Get started!
3. Add the trigger filters and conditions to target the cohort determined in step 1.
4. Set the incentive or offer. 
5. Modify the notifications using Recharge notifications.
6. Update the A/B test percentages as needed. A 50/50 split will reach statistical significance faster than other splits, but the percentages can be updated based on your business needs.
7. Save and click Activate Flow. 

Step 3 - Review results

To determine the success of the incentive or offer, check out your branch results within the flow. Easily see how many customers entered the flow and how many accepted the offer and processed their next charge.

There will naturally be a delay in when the next charge branch results will be complete. You must wait until the customer reaches their next charge, so if the charge frequency is every 2 months, there will be at least a 2-month delay between when a customer enters the flow and completes their next charge. This timeframe is further delayed if a customer delays their next charge date.

Once you determine if the incentive or offer was successful, consider creating an ongoing “Basic Flow” to apply the incentive or offer to 100% of the customer segment.