In recent years of web development A/B Testing has been used by marketers to test different variations of pages and components (vertically arranged content blocks). Testing different versions of components – let’s say two menu sections – helps determine which variation is optimal for the specific goal that the company wants to achieve.
There are a couple of 3rd party services that can be integrated into WordPress. However they didn’t fulfill our standards. That’s why we came to the conclusion to develop a custom solution. So far it’s based on our self-developed WordPress Theme Flynt, but we keep working on making it available for every WordPress page.
To be honest, A/B testing is a complicated feature to implement. But we have achieved the most important goal: our solution meets all the required criteria that 3rd party tools could not. Learn here, how we came to the result and what could be improved.
Pros & Cons of 3rd party A/B testing tools
Before we get to what the pros and cons of third-party A/B testing tools are, let’s get an idea of what a good A/B test should look like. The steps are:
- Visitor lands on a webpage.
- We ask for visitor’s cookie consent.
- If the visitor accepts, the A/B testing variant is saved on a cookie or localStorage to persist it on multiple visits.
- Visitor’s actions are tracked based on the goals of the test.
Intuitive UI as major advantage
One major advantage that most of these tools have, is their intuitive UI. It helps the content manager and marketers manipulate the content, set goals and run tests without the need of a development team.
Cons: Loading external script cause to problems
However, most of them require you to load an external script that will determine the variant to display and then manipulate the content of the page accordingly.
- GDPR-compliance: In order to run A/B testing you need the visitor’s consent, which means that you cannot load the script until you get it.
- Flashing content: If you get consent you load the script and the A/B testing tool swaps the content based on the variant that is selected. This can lead to flash of original content (FOOC) and a bad visitor’s experience.
- Load time: External scripts can also lead to significantly increased load time for your page.
Problems with 3rd party tools
In the case of one of our partner’s websites we identified the following issues of their current A/B testing tool:
- Huge page load times (3s – 4s).
- Current tool isn’t fully GDPR-compliant.
- Extra cost for tool’s license.
- Multiple tools to create content (main website uses Flynt while A/B testing pages are created in an external tool).
We started researching alternative tools, but they led to potential conflicts like FOOC, GDPR incompatibility and SEO issues. It turned out no 3rd party would provide a solution that matches with the high demands we and the client have set for a technical solution. That’s why we decided to develop it on our own.
The challenge: Custom A/B testing with page templates
First approach: Creating the content only server side
One big drawback with this approach is that page caching is not possible, as every request would have to go through the server to determine what to render. This could lead to potential page speed problems and unnecessary requests. Increased costs and potentially server downtime could be the consequences.
Second approach: Using two different URLs
As a second approach we examine the idea of having completely different pages for the A/B testing and either redirect from the ad to the different URL or choose one from a server request. While this approach could (partially) eliminate the caching problem, it creates other issues.
In this case we would have two or more pages with almost identical content that would severely penalize the SEO for the website if we were not careful.
Working solution: Creating a page template
Our current solution manages to eliminate most of the described problems: you can switch between page variants and content is exchanged without flickering and only with minimal Cumulative Layout Shift (CLS).
What we end up doing is creating a Custom Page Template from the page that holds the default variation as its content. This way we can easily clone the page to create alternative variations that we want to test.
How it works
We set the page variations to redirect to the default variant. This way, even though we have multiple pages, all pages lead to a single url. This helps us to also make sure that it is easy to switch the default page if, when the test is done, we want to use a different variation as our page.
When the A/B testing page is fully rendered we check if the current visitor has accepted the cookies. If so, we save the selected variant in cookies in order to deliver the same experience in consecutive visits and get accurate data for the test. Finally, we execute any business logic and trigger any event we want to report in the analytics platform of our choice (e.g. Google Analytics, Google Tag Manager, etc.).
We also save the A/B testing page and variation to a category in order for our partner to easily filter and preview them.
As an extra feature, we added the option to set the variant – using UTM parameters – from the URL. For example, to direct visitors to different variants of an ad campaign or a call-to-action.
While the above solution might be good enough, it is not perfect. Ideally we would
- avoid duplicate content for testing an alternative image, button color or the order of components.
- use A/B testing components instead of pages to see if they can be more flexible.
To sum up, let’s recap what we have achieved with the custom solution for the client’s website:
- Reduced page load times to less than 1s.
- A/B Test landing pages work within the WordPress Theme Flynt.
- No flickering or FOOC.
- GDPR-compliant testing.
We would say that A/B testing is a complicated feature to implement and with different solutions for different cases. It is definitely something that needs to be performant and have the least amount of overhead on a website as possible, while also respecting the privacy of the visitor.
Have you ever done any A/B Testing yourself? Do you have any thoughts or ideas on how we could further develop A/B Testing? Share this article and do not hesitate to reach out!