How to implement custom A/B testing in WordPress?

In recent years of web development A/B testing in WordPress and other CMS has been used by marketers to test different variations of pages and components (vertically arranged content blocks). Testing different versions of components – let’s say two menu sections – helps determine which variation is optimal for the specific goal that the company wants to achieve.

There are a couple of 3rd party services that can be integrated into WordPress. However they didn’t fulfill our standards. That’s why we came to the conclusion to develop a custom solution. So far it’s based on our self-developed WordPress Theme Flynt, but we keep working on making it available for every WordPress page.

To be honest, A/B testing is a complicated feature to implement. But we have achieved the most important goal: our solution meets all the required criteria that 3rd party tools could not. Learn here, how we came to the result and what could be improved.

Pros & Cons of 3rd party A/B testing tools

Before we get to what the pros and cons of third-party A/B testing tools are, let’s get an idea of what a good A/B test should look like. The steps are:

1. Visitor lands on a webpage.
2. We ask for visitor’s cookie consent.
3. If the visitor accepts, the A/B testing variant is saved on a cookie or localStorage to persist it on multiple visits.
4. Visitor’s actions are tracked based on the goals of the test.

Intuitive UI as major advantage

One major advantage that most of these tools have, is their intuitive UI. It helps the content manager and marketers manipulate the content, set goals and run tests without the need of a development team.

Cons: Loading external script cause to problems

However, most of them require you to load an external script that will determine the variant to display and then manipulate the content of the page accordingly.

• GDPR-compliance: In order to run A/B testing you need the visitor’s consent, which means that you cannot load the script until you get it.
• Flashing content: If you get consent you load the script and the A/B testing tool swaps the content based on the variant that is selected. This can lead to flash of original content (FOOC) and a bad visitor’s experience.
• Load time: External scripts can also lead to significantly increased load time for your page.

Problems with 3rd party tools

In the case of one of our partner’s websites we identified the following issues of their current A/B testing tool:

• Huge page load times (3s – 4s).
• Current tool isn’t fully GDPR-compliant.
• Extra cost for tool’s license.
• Multiple tools to create content (main website uses Flynt while A/B testing pages are created in an external tool).

We started researching alternative tools, but they led to potential conflicts like FOOC, GDPR incompatibility and SEO issues. It turned out no 3rd party would provide a solution that matches with the high demands we and the client have set for a technical solution. That’s why we decided to develop it on our own.

The challenge: Custom A/B testing in WordPress with page templates

First approach: Creating the content only server side

Our initial thought was to remove as much workload as possible from the browser and do all the selection and rendering of the content on the server. That would eliminate FOOC and flickering completely, and it would also ensure that no javascript would have to run to swap or manipulate the content.

One big drawback with this approach is that page caching is not possible, as every request would have to go through the server to determine what to render. This could lead to potential page speed problems and unnecessary requests. Increased costs and potentially server downtime could be the consequences.

Second approach: Using two different URLs

As a second approach we examine the idea of having completely different pages for the A/B testing and either redirect from the ad to the different URL or choose one from a server request. While this approach could (partially) eliminate the caching problem, it creates other issues.

In this case we would have two or more pages with almost identical content that would severely penalize the SEO for the website if we were not careful.

Working solution: Creating a page template

Our current solution manages to eliminate most of the described problems: you can switch between page variants and content is exchanged without flickering and only with minimal Cumulative Layout Shift (CLS).

What we end up doing is creating a Custom Page Template from the page that holds the default variation as its content. This way we can easily clone the page to create alternative variations that we want to test.

How it works

We set the page variations to redirect to the default variant. This way, even though we have multiple pages, all pages lead to a single url. This helps us to also make sure that it is easy to switch the default page if, when the test is done, we want to use a different variation as our page.

When the A/B testing page is fully rendered we check if the current visitor has accepted the cookies. If so, we save the selected variant in cookies in order to deliver the same experience in consecutive visits and get accurate data for the test. Finally, we execute any business logic and trigger any event we want to report in the analytics platform of our choice (e.g. Google Analytics, Google Tag Manager, etc.).

Additional features

We also save the A/B testing page and variation to a category in order for our partner to easily filter and preview them.

As an extra feature, we added the option to set the variant – using UTM parameters – from the URL. For example, to direct visitors to different variants of an ad campaign or a call-to-action.

Possible Improvements

While the above solution might be good enough, it is not perfect. For the ideal A/B testing in WordPress we would

1. avoid duplicate content for testing an alternative image, button color or the order of components.
2. use A/B testing components instead of pages to see if they can be more flexible.

Conclusion: A/B testing in WordPress

To sum up, let’s recap what we have achieved with the custom solution for the client’s website:

• Reduced page load times to less than 1s.
• A/B test landing pages work within the WordPress Theme Flynt.
• No flickering or FOOC.
• GDPR-compliant testing.

We would say that A/B testing in WordPress is a complicated feature to implement and with different solutions for different cases. It is definitely something that needs to be performant and have the least amount of overhead on a website as possible, while also respecting the privacy of the visitor.

Have you ever done any A/B testing yourself? Do you have any thoughts or ideas on how we could further develop A/B testing in WordPress? Share this article and do not hesitate to reach out!