Multivariate Testing

Multivariate testing is a process for testing hypotheses where more than one variable is changed. Unlike a traditional A/B test, a multivariate test can involve multiple factors changing at the same time. The goal of multivariate testing is to find the most effective combination of variables from all possible options.

In email marketing, multivariate testing involves changing variables like: send time, subject line, images, content of email, colors, tone, and presence of links. The results of each combination are recorded and analyzed to determine the most effective variation.

How is multivariate testing different from A/B testing?

Using some of the same mechanics of A/B testing, multivariate testing often compares more than 1 variable at a time and reveals more about how those different variables interact and influence one another. Similary to A/B testing, traffic is still split between different pages.

How do you do a multivariate test?

1. Identify a problem to solve.
2. Create a hypothesis about how to solve the problem
3. Create variations that may solve the problem
4. Determine an appropriate sample side
5. Test tools to make sure they're working
6. Send traffic to the multivariate testing pages
7. Analyze results
8. Revise your hypothesis

When would you use a multivariate test?

You would use a multivariate test when looking to optimize smaller elements on a page. You will be able to test the effect of changing single elements, as well as be able to assess the combined interaction effect of several element changes.