What’s better? Multivariate or A/B Testing?

Posted on April 7, 2016 by Loyalty Builders

loyalty-button

At the very simplest level, testing amounts to asking your customers a question. Which headline works? Which color kills? What price opens wallets? Canny marketers have known for more than 100 years that testing is absolutely the best way to refine and perfect the performance of advertising and marketing campaigns.

Mostly they’ve relied A/B tests that answer one question at a time. Multivariate testing, however, is many times more powerful. It’s like a questionnaire that asks multiple questions and serves up multiple answers from a single test. Multivariate tests deliver more information faster, and cost way less than answering the same questions with multiple A/B splits.

Testing multiple factors is obviously more complex than simpler A/B tests. So we’re now offering our years of experience in a testing service that does the mathematical heavy lifting for you. For a modest fee you now have a powerful a la carte testing capability at your command. Details below.

1. What is the Multivariate Testing Service?

A service to design and interpret the results of multivariate tests of marketing campaigns.

2. What is the difference between multivariate testing and A/B testing?

  • In A/B testing (also called ‘split-run’ or ‘one factor at a time), only one variable is changed in each test. To test multiple variables (for example, discount, subject line, background color in an email), multiple tests are needed.
  • With multivariate testing, multiple variables can be tested simultaneously. Multivariate testing takes less time and costs less money. It is often the only feasible way to test multiple factors.

3. What is the process?

  • You create your campaign collateral and decide which factors you want to test
  • You send us your customer transaction data
  • We design multivariate tests for your marketing campaigns
  • We pick the control groups and decide which groups get what material
  • You send your direct marketing campaigns and track the sales
  • You send us more transaction data after campaign ends
  • We analyze, interpret, and report results
  • We tell you which variants work best
  • We recommend new tests and marketing campaign improvements

4. What can be tested?

  • Targeting
  • Offers
  • Subject lines
  • Other collateral components

5. What you will learn

  • Which campaign elements most affect campaign performance, for each campaign tested
  • Of the elements tested, which variation is most effective
  • How to field the most effective campaigns to get the highest response rates

6. Who we do it for

Anyone who is running a direct marketing campaign.

7. What does it cost?

  • One test per month, with control groups – $2,000/month
  • Two tests per month – $3,000/month
  • Three tests per month – $3,750/month
  • One free test per month included in Proof-of-Concept trials
  • No long term contract required—hire us for one month or many months
  • Discounts for existing Loyalty Builders pilot or extended subscription clients

8. What you do

  • Create the campaign collateral
  • Send out the campaigns with the tests (email or print)
  • Send transaction data to Loyalty Builders each month while tests are running

If you believe as deeply as we do that testing is the key to better marketing results, the value here should be obvious. If you see the logic but need persuading, let us help you test multiple factors just one time. You’ll be convinced. We’ve been doing multivariate testing for more than a decade. We know it’s fast. We know it saves. We know it works.

If you believe as deeply as we do that testing is the key to better marketing results, the value here should be obvious. If you see the logic but need persuading, let us help you test multiple factors just one time. You’ll be convinced. We’ve been doing multivariate testing for more than a decade. We know it’s fast. We know it saves. We know it works.

Tags :