All Articles

How to A/B test your React and Gatsby components

a-b-testing-react-gatsby-mixpanel

React and its components structure is a excellent starting point to A/B test your apps. In this article I will explain how to implement A/B tests in your React/Gatsby app. And how to collect the experiment data with Mixpanel.

What to A/B test?

I’m not going to explain A/B testing in-depth. The concept if fairly simple:

  1. Research analytics and think about goals (eCommence sites most likely want to increase their conversion rate, I on the other hand want to increase my newsletter sign up rate)
  2. Create a hypothesis (E.g. I think my newsletter signups will increase if the sign up form will pop-up)
  3. Create a variation (Create a extra newsletter sign up component that pop ups after 15 seconds)
  4. Test both variants (by routing users to one or the orther component)
  5. Analyzing results and draw conclusions

If you want to learn more about A/B testing, check out this guide from VWO.


Create the variation component

Before we start including A/B testing packages in our project, I need a component to test. This blog will function as test case, I will create a pop up newsletter sign up component (I know I’m sorry).

I already have a MailSubscribe.jsx compontent. I duplicated the component and wrapped it inside a modal. This post isn’t about how you make a popup mail subscribe component, but if you are curious you can check out the code @ Gitlab. The only thing you need to know is that we now have two components: MailSubscribe.jsx and MailSubscribePopUp.jsx. In the next step we’ll learn how to create a 50/50 split between the components.


Set up A/B testing

To spread out users between the two components, I use @marvelapp/react-ab-test. The library wraps two components in a Experiment component. The components you want to test, will go inside the Variant component. For example:

<Experiment name="My Example">
    <Variant name="A">
        <div>Version A</div>
    </Variant>
    <Variant name="B">
        <div>Version B</div>
    </Variant>
</Experiment>

Seems easy right? Let’s install the library.

yarn add @marvelapp/react-ab-test

Then I wrap my two mailsubscribe components inside my Post.jsx component.

import MailSubscribe from '../MailSubscribe'
import MailSubscribePopUp from '../MailSubscribePopUp/MailSubscribePopUp'
import { Experiment, Variant } from 'react-ab-test'

const Post = ({ post }) => {
    const { tags, title, date } = post.frontmatter
    const { html } = post
    const { tagSlugs } = post.fields

    return (
        <Experiment name="newsletter_popup">
            <Variant name="original">
                <MailSubscribe />
            </Variant>
            <Variant name="popup">
                <MailSubscribePopUp />
            </Variant>
        </Experiment>
    )
}

export default Post

So let’s test this, we know it works when we see only one of the two components. It works! Only one of the two components is visible.

React popup component

I can also check localstorage if the package succesfully stored the test data. This data is to prevent that a user will get two variations on same browser. And there it is, I have the popup variant.

React popup component console


Collecting the results with Mixpanel

Technically we are ready for our first A/B test, but without data collection we won’t know what variant performs better. There are a lot of tools to collect this data, in this tutorial I’m using Mixpanel, but feel free to use a other tool or to collect your own data. First we need to embed the Mixpanel script in our website.

After making a account, the embed script is direct visible. We only need the unique code at the end of the script.

Mixpanel welcome screen

Install the Mixpanel package.

yarn add mixpanel-browser

Create a file for Mixpanel, like mixpanel.js. In that file we need initialize the Mixpanel function and export that function. The token that you need is somewhere inside the Mixpanel script tag.

mixpanel.js:

import mixpanel from 'mixpanel-browser'

export const _ = mixpanel.init('REDACTED')

Import the _ variable in your main component. And the last thing we need to do is to test our connection to Mixpanel. Add the following code to a component, it doesn’t matter where, as long as it runs.

import mixpanel from 'mixpanel-browser'
//Fire the function inside a component
mixpanel.track('test')

If successful, you will receive a notification in your Mixpanel dashboard.

Success emit event to Mixpanel


Add Mixpanel success case to our A/B test

The last step is to let Mixpanel know when our goal (a sign up for my newsletter) is triggered. Use the mixpanel.track() function to signal when a ‘win case’ is hit. This is my MailSubscribePopUp.jsx component

import React, { useState, useEffect } from 'react'
import MailchimpSubscribe from 'react-mailchimp-subscribe'
import Formsy from 'formsy-react'
import Input from '../../UX/Input'
import styles from './MailSubscribePopUp.module.scss'
import Modal from '../../UX/Modal/Modal'
import mixpanel from 'mixpanel-browser'

const url = process.env.MAILCHIMP_CAMPAIGN_URL

const MailSubscribePopUp = () => {
    const [showModal, setShowModal] = useState(false)
    const [canSubmit, setCanSubmit] = useState(false)
    const disableButton = () => setCanSubmit(false)
    const enableButton = () => setCanSubmit(true)

    const submit = (model, subscribe) => {
        subscribe({ EMAIL: model.email })
        mixpanel.track('Experiment: Newsletter popup -> win', {
            Experiment: 'newsletter_popup',
            Variant: 'popup',
        })
    }

    useEffect(() => {
        mixpanel.track('Experiment: Newsletter popup -> start', {
            Experiment: 'newsletter_popup',
            Variant: 'popup',
        })
        timeoutModal()
    }, [])

    const timeoutModal = () => {
        setTimeout(() => {
            setShowModal(true)
        }, 1000 * 15)
    }

    return (
        <Modal
            title="Sign up for my newsletter!"
            show={showModal}
            close={() => setShowModal(false)}
        >
            <MailchimpSubscribe
                url={url}
                render={({ subscribe, status, message }) => (
                    <Formsy
                        onValidSubmit={model => submit(model, subscribe)}
                        onValid={enableButton}
                        onInvalid={disableButton}
                    >
                        <p>
                            Want more articles? Subscribe to my newsletter and I
                            will keep you updated about future content!
                        </p>
                        <div className={styles['form-con']}>
                            <Input
                                className={styles['input']}
                                name="email"
                                placeholder="emailaddress"
                                validations="isEmail"
                                //validationError="This is not a valid email"
                                required
                            />
                            {status === 'success' ? (
                                <button
                                    className={
                                        styles['button'] +
                                        ' ' +
                                        styles['button__success']
                                    }
                                    succes={true}
                                    type="button"
                                    disabled={true}
                                >
                                    Thanks for subbing!
                                </button>
                            ) : (
                                <button
                                    className={styles['button']}
                                    type="submit"
                                    disabled={!canSubmit}
                                >
                                    Sign up!
                                </button>
                            )}
                        </div>
                    </Formsy>
                )}
            />
        </Modal>
    )
}

export default MailSubscribePopUp

And MailSubscribe.jsx:

import React, { useState, useEffect } from 'react'
import MailchimpSubscribe from 'react-mailchimp-subscribe'
import Formsy from 'formsy-react'
import Input from '../../UX/Input'
import styles from './MailSubscribe.module.scss'
import mixpanel from 'mixpanel-browser'

const url = process.env.MAILCHIMP_CAMPAIGN_URL

const MailSubscribe = () => {
    const [canSubmit, setCanSubmit] = useState(false)
    const disableButton = () => setCanSubmit(false)
    const enableButton = () => setCanSubmit(true)

    const submit = (model, subscribe) => {
        subscribe({ EMAIL: model.email })
        mixpanel.track('Experiment: Newsletter popup -> win', {
            Experiment: 'newsletter_popup',
            Variant: 'original',
        })
    }

    useEffect(() => {
        mixpanel.track('Experiment: Newsletter popup -> start', {
            Experiment: 'newsletter_popup',
            Variant: 'original',
        })
    }, [])

    return (
        <MailchimpSubscribe
            url={url}
            render={({ subscribe, status, message }) => (
                <Formsy
                    onValidSubmit={model => submit(model, subscribe)}
                    onValid={enableButton}
                    onInvalid={disableButton}
                >
                    <p>
                        Want more articles? Subscribe to my newsletter and I
                        will keep you updated about future content!
                    </p>
                    <div className={styles['form-con']}>
                        <Input
                            className={styles['input']}
                            name="email"
                            placeholder="emailaddress"
                            validations="isEmail"
                            //validationError="This is not a valid email"
                            required
                        />
                        {status === 'success' ? (
                            <button
                                className={
                                    styles['button'] +
                                    ' ' +
                                    styles['button__success']
                                }
                                succes={true}
                                type="button"
                                disabled={true}
                            >
                                Thanks for subbing!
                            </button>
                        ) : (
                            <button
                                className={styles['button']}
                                type="submit"
                                disabled={!canSubmit}
                            >
                                Sign up!
                            </button>
                        )}
                    </div>
                </Formsy>
            )}
        />
    )
}

export default MailSubscribe

Check if the results make it into Mixpanel:

Mixpanel events


Analyze the results

And like that, we are A/B testing! Next up is to generate some traffic to your experiment and analyze the results. The A/B testing functionality in Mixpanel is for paying customers only, but with two custom funnels we can get a overview of how our experiment is going. Check out my setup:

Mixpanel funnel original

Mixpanel funnel variant

And the dashboard: Mixpanel funnel variant

There are two funnels, with each two steps. The first step is all the users that trigger the experiment, the second step are the users that convert to the goal of the test. If the conversion ratio of our popup variant out preforms the original variant, I can safely assume that I made my website a little bit better.

I’m really curious to see what you can do with A/B testing! If I can help let me know.

Header image credit: Starline