Old School vs. New School Testing
David Baker has a good column in Email Insider this week, What Really Makes An Email Program Click. In it, he writes “Remember one thing: You test to prove or disprove your hypothesis, not just for the exercise.”
To me, that sounds like old school testing, where one ‘thing’, one hypothesis, is tested. Formally, that is called split-run or A/B testing. There is only one variable, and two variations of it. It works, but it is time consuming and (if there is more than one hypothesis) needs lots of test subjects and money. Consequently many companies just don’t do it, relying instead on their intuition.
Today however you can test many variables simultaneously, and even uncover the interactions between the variables. The two main types of marketing tests are testing offers and testing segments. Examples of offer tests are HTML vs. text, blue background vs. red, $10 off on $100 purchase vs. $20 off on $200, etc. Examples of segment tests are recent purchasers, potential defectors, some geography, various loyalty deciles, etc. The math is more complicated (it’s called Factorial Design), but there are vendors to do it for you, and the results are outstanding.
So yes, absolutely test. An email test is generally over within two days, and it is one of the two best ways to improve your marketing campaigns. But test everything, not just ‘your hypothesis.’