‘Tis is the season for new customer acquisition and increased revenue! But did you know it’s also a great opportunity for some strategic A/B testing? Yes! You can get even more value out of holiday traffic! But, as with most things, there is a right way and a wrong way to go about it. What to test and what not to test? Following is a guide to what factors we think are important to consider, as well as some recommendations for the best (aka most effective and least disruptive) areas to focus your testing efforts during the holiday season.
You know it: holiday is crunch time
The holiday season can be very profitable for merchants. This year, total holiday spending is still expected to be high, growing 3-4% to as much as $966 billion. Online sales are projected to grow even more, increasing 4.8% over 2022.
Therefore, before you do anything else, make sure your eCommerce site is ready for the uptick in holiday traffic. Test your site ahead of November to make sure your site speed is optimized, the shopping funnel isn’t obstructed, and that customers can easily find what they want and will have all of the information they need to reduce risk and decide to purchase (e.g., return policy, sizing, shipping estimates, etc.).
To even consider A/B testing during the holiday season, these basics must be in place first.
The holidays are crazy. Why would I even consider testing then?
We get it. There’s a lot to manage during the holiday season. But, if you plan ahead (and maybe work with your agency) to schedule some tests, you may find the results very worthwhile – and profitable!
Get results faster
We recommend certain A/B testing during the holidays because tests can reach “statistical significance” more quickly due to higher traffic. Reaching statistical significance ensures that you can have confidence that the results of the test are based on enough data to show that the “winning” variation isn’t a mistake or based on chance.
With the increased traffic coming to your site, you can apply insights faster – even getting a small increase in conversion rate from a winning variation could make a big difference to your bottom line.
Understand new customers
If you’re like most merchants, you’re running advertising during the holidays. This results in (hopefully!) more sales but also means that many of the shoppers on your site are probably different from your regular customers who shop with you year-round. For example, they are more likely to be first-time visitors who are unfamiliar with your brand and/or catalog of products. They may be more discount-driven – comparing prices to find just the “right” gift. Also, they are less likely to be shopping for themselves, so they may not be your target customer at all.
Testing different marketing content and promotions with these new customers can provide all kinds of insights that you can use for retargeting campaigns or to inform more effective holiday marketing strategies next year.
What not to test: 3 holiday A/B testing no-no’s
In a nutshell: don’t test anything that could disrupt the sales funnel and cause shoppers to drop off or abandon cart. More specifically, we recommend staying away from CRO testing the following three areas during the holidays.
1. Usability of new features
The key to conversion during holiday is: the less friction, the better. This is true from product findability to streamlining checkout. Therefore, help out your returning customers – who probably have a high intent to buy – and ensure they come back to a familiar shopping experience so they can purchase quickly and efficiently.
Avoid testing features like new primary buttons, new steps in add-to-cart, and new page layouts. Also, don’t touch menus, search or filters – you’ve trained your customers to use these in a certain way. Instead, add new features usability testing to your spring testing roadmap.
2. Information architecture
Again, anything that may make it more difficult for returning users to find what they’re looking for – such as a reorganization of your product catalog – is a definite ‘no.’ This includes testing a new menu, navigation headers, search results pages, and more.
By all means, add a holiday gift guide! But don’t change the structure of your existing main menu. Also, it’s a best practice to use the same product categories for the gift guide that you do on the site for a cohesive, familiar experience.
3. Customer account management functionality
We recommend not testing new customer account features during the holiday season. There’s often a spike in customers logging in to redeem loyalty points, review past purchases or modify their address book. For subscriptions, customers may want to pause deliveries during holiday travel or increase quantities to ensure there’s enough coffee or soap for house guests. The last thing you want to do is confuse and frustrate customers who may have high purchase intent or feel pressed for time.
What you should A/B test: 5 opportunities for insights
The holiday season is a prime opportunity to test what promotions or marketing content works to capture and convert new customers. The key is to find a balance of what’s going to resonate with new customers without becoming a friction point for returning customers.
1. Promotions
Perhaps the test that will give you the biggest bang for your buck is to determine the lowest discount threshold that will still increase conversions. For example, if a 15% discount is as effective as a 20% discount for your margins, that 5% difference could be significant for your margins and profits during the holiday season. Testing can help you determine what is the lowest bar for the highest conversion rate. Additionally, test what other promotions, like free shipping thresholds or product bundles are most effective for converting first-time visitors and returning customers.
2. Marketing channels
Increased traffic during this time, as well as (maybe) more budget allocated for holiday marketing efforts, makes the holidays a great time to test new channels, including social media, SMS and paid search. Make sure to monitor closely so that if a channel isn’t performing as you need it to, you can quickly pivot.
3. Marketing content
While returning customers are definitely your VIPs, holiday is also a great time to boost acquisition efforts. Holiday visitors are likely different from your core audience. What do they want to know about your products? Do you need to educate them about your brand? Do they respond best to humor? FOMO? Social proof? Even if new customers don’t convert, testing different marketing content can give you insights into what works and what doesn’t when you remarket to new customers after holiday.
4. Visual assets
As a subset of marketing content, don’t forget to test different visuals, such as product photos versus lifestyle imagery to see what drives higher engagement.
5. Email collecting pop-ups/drawers
Many holiday shoppers are looking for deals and will be more receptive to giving their email address to receive a discount. Therefore, holiday can be a great time to test where on the site and at what stage in the sales funnel to display these offers.
Hot tip: Leverage dynamic traffic allocation
Make sure you get the most value out of the winning test variant! Most testing platforms, including VWO and Optimizely, enable merchants to more quickly pivot to the winning variant. Merchants can configure test settings to automatically assign more traffic to the test variant showing a higher likelihood to convert (once the test reaches statistical significance).
Be strategic with testing and plan ahead
We think it’s important to always be testing to optimize a site’s performance and find ways to increase conversions. But that doesn’t mean test everything all the time. We work with clients to strategically develop a quarterly and annual testing roadmap that supports their business goals. We plan specific tests year-round with altered tests for holiday traffic during November and December.
Here's what this process typically looks like:
1. Develop an A/B testing roadmap
Plan tests alongside ongoing development throughout each quarter and year. This will ensure that testing doesn’t interfere with deployments of new features or other changes on the site.
2. Set up each test’s goals, hypotheses, and metrics
Determine clear goals that support your business objectives, such as increasing revenue. Also, outline how you think the changes will affect the goals. For example, a hypothesis you want to test could be that raising the free shipping threshold to $50 will increase AOV. Lastly, determine which metrics you will use to measure progress toward your goals.
3. Integrate the test platform on site
Connect goals/metrics on site using your tool of choice, such as Google Analytics metrics, a custom event created in Google Tag Manager (GTM), or new in-app goals (e.g., user experiences Variant A, then returns to purchase later) for each scheduled test.
4. Deploy testing one of two ways
One way to deploy testing is to design/develop in your regular workflow and set up a test variant in a non-live Shopify theme to preview it before it goes live. Another way is to set up the test variant in-app (e.g., VWO) using the app’s code or visual editor. This enables the app to automatically insert the variant code when the test is triggered. Tests should run 10-14 days, unless statistical significance is reached earlier, which could occur with elevated holiday traffic.
5. Deploy the winning variant!
Review the results of the test. Did you reach your goals? Was your hypothesis correct? What did you learn? Based on these insights, you can decide to deploy the variant that had the best results or try again. Interested in knowing more about how to optimize your Shopify eCommerce site? We’re happy to chat about CRO, user testing, UX site audits, site speed improvements and more – contact us!