Multivariate Testing Explained... For Dummies

I test con­stantly in an effort to boost my response rates. So it’s no won­der why some peo­ple have asked me about mul­ti­vari­ate test­ing. Specif­i­cally, they want to know what is it, how does it work, and what’s “Taguchi.”

First of all, Taguchi is just one form of mul­ti­vari­ate test­ing. I’m not a math­e­mati­cian. And Dr. Genichi Taguchi and his work specif­i­cally relate to the man­u­fac­tur­ing indus­try, which was later proven ben­e­fi­cial in the car industry.

Not direct mar­ket­ing. (At least, not at first.)

Dr. Taguchi’s for­mula was only recently extrap­o­lated to the areas of direct mar­ket­ing, adver­tis­ing, and now Inter­net mar­ket­ing, and has become the basis behind mul­ti­vari­ate test­ing and the pop­u­lar­iza­tion of mul­ti­vari­ate test­ing as a whole.

More often than not, the name “Taguchi” is a buzz­word often bandied about, even by peo­ple who’s mul­ti­vari­ate tests do not use Taguchi’s method specif­i­cally. The for­mula is rather com­plex, and I’ll leave it to those more capa­ble than me to explain it to you.

But in this arti­cle, let me explain the basics of mul­ti­vari­ate test­ing, using layman’s terms and as best as I can. I hope not to lose you along the way. Fin­gers crossed.

What mul­ti­vari­ate test­ing is, is this…

Split-​​tests are nor­mally based on two dif­fer­ent ver­sions of one ad. That’s why they’re often called “A/​B Split Tests” (or “A/​B Split Runs”). The object is to deter­mine which ad — either ver­sion “A” or “B” of a saleslet­ter, for exam­ple — pulls the great­est response.

Nor­mally, mar­keters test one thing at a time: whether it’s the head­line, the price, the offer, the copy, the color, the guar­an­tee, or what­ever. Each of these is a “variable.”

In direct mail, for instance, mar­keters will send a small run of ver­sion “A” to a spe­cific num­ber of peo­ple, and ver­sion “B” (often, this is done simul­ta­ne­ously) to another group. Each ver­sion will test one vari­able at a time to deter­mine which ver­sion pulls the best.

You then take the win­ning ad (i.e., the one with the high­est con­ver­sion of leads to sales), which becomes what is called the “con­trol,” and use it to mail to the rest of the list.

Depend­ing on how big your list is, you can do a vari­ety of these split-​​tests before you deter­mine the con­trol and launch your win­ning ad. (As you can see, that can be limiting.)

Online, it’s a lit­tle more effec­tive, because you can have a pro­gram that ran­domly pulls either one of these two ver­sions with each vis­i­tor, and have it cal­cu­late the results after reach­ing a spe­cific sam­ple size (i.e., a cer­tain num­ber of vis­i­tors and/​or sales)

But here’s the dilemma — or the lim­i­ta­tion, in other words.

Split-​​tests are lin­ear in fashion.

With your ad, you can test two vari­a­tions of one par­tic­u­lar vari­able to deter­mine the win­ning ver­sion. Once you’re done, you take the win­ning ver­sion, choose a new vari­able, and run a sub­se­quent test. You then move on to the next test. Rinse and repeat.

The object is to con­stantly beat your control.

Objec­tively, you can only test only one vari­able at a time so that you can pre­cisely deter­mine what caused a boost in con­ver­sion. If you tested mul­ti­ple vari­ables simul­ta­ne­ously, you would be at a loss as to which vari­able actu­ally caused the jump.

With a long saleslet­ter, you have hun­dreds of pos­si­ble vari­ables, too. You can test head­lines, lead copy, deck copy, col­ors, pic­tures, posi­tions, lay­outs, prices, pre­mi­ums, offers, order forms, order but­tons, cap­tions, bul­lets, guar­an­tees, tes­ti­mo­ni­als, ad nauseum.

But that’s not all. You also have mul­ti­ple vari­a­tions of each vari­able — such as dif­fer­ent ver­sions of a head­line. (Vari­a­tions of one vari­able is called a “fac­tor.”) So in these cases, you can run a 3-​​, 5-​​, or even 10-​​way split test (e.g., 10 dif­fer­ent headlines).

Call it an “A/​B/​C/​D/​E, etc” split-​​test, in other words.

But that’s not the prob­lem. In tra­di­tional split-​​tests, you typ­i­cally test one fac­tor at a time (dif­fer­ent ver­sions of one vari­able). Each test is done indi­vid­u­ally. And sequen­tially.

With sequen­tial split-​​tests, how­ever, the prob­lem is that it’s pos­si­ble a vari­able in a sub­se­quent test could have pro­duced a higher response with the loser of a pre­vi­ous test.

Sounds crazy, I know. But let me give you an exam­ple to illustrate.

For instance, you test two dif­fer­ent ver­sions of a head­line. After a num­ber of sales, you deter­mine that head­line “A” out­pulled head­line “B.” So nat­u­rally, you con­clude that the copy with head­line “A” has become your con­trol, and you move on to the next test.

In a sub­se­quent split-​​test, you decide on test­ing two dif­fer­ent prices. After a while, your test has deter­mined that price “B” is the win­ner and con­verts more sales. This means that your new con­trol is now the copy made up of head­line “A” and price “B.”

Fol­low­ing me so far?

A ques­tion remains: what would hap­pen if you tested prices with the loser from the pre­vi­ous split-​​test — that is, test­ing the two dif­fer­ent prices with head­line “B?” Would price “B” still be the win­ner? The response could have been greater. Or maybe not.

But the prob­lem is, you don’t know.

Sure, chances are remote that the los­ing head­line would affect the price test. And sure, you can go back and test prices with the los­ing head­line if you want to be sure.

But the prob­lem is when tests become more com­plex. For exam­ple, what hap­pens if you want to run mul­ti­ple split-​​tests with other vari­ables, like guar­an­tees or bonuses? What hap­pens if you want to test sev­eral vari­a­tions of each vari­able in each test?

If you want to run more split-​​tests to test other vari­ables, with fac­tors as large as up to 10 vari­a­tions per vari­able or more (like, say, eight dif­fer­ent head­lines, five dif­fer­ent prices, four dif­fer­ent guar­an­tees, nine dif­fer­ent offers, 12 dif­fer­ent prod­uct pic­tures, etc)…

… Then you can see how lim­it­ing such tests can be if you do it in a sequen­tial fashion.

For one, the time it would take to run each test would take for­ever to run them all — let alone going back and test­ing sub­se­quent win­ners with losers from pre­vi­ous tests. And sec­ond, the poten­tial per­mu­ta­tions become con­sid­er­ably vast and more complicated.

That’s the flaw with tra­di­tional split-​​testing. It’s lin­ear, you can only test one vari­able at a time, it takes a heck­u­valot of time, and it doesn’t test dif­fer­ent per­mu­ta­tions — or, in other words, dif­fer­ent “com­bi­na­tions” — between all the dif­fer­ent vari­ables and factors.

This is where mul­ti­vari­ate test­ing — and soft­ware based on it — comes in. The goal is to dis­cover the best pos­si­ble com­bi­na­tion of fac­tors that out­pulls all pos­si­ble combinations.

(I hope you’re still fol­low­ing me.)

Of course, it won’t deter­mine this based on test­ing every­thing. After all, not a lot of peo­ple can drive that num­ber of traf­fic, any­way. But with sophis­ti­cated soft­ware nowa­days, it can make fairly good pre­dic­tions based on the sci­ence of prob­a­bil­i­ties.

The soft­ware can sci­en­tif­i­cally “guess” an opti­mal com­bi­na­tion of variables…

… Simul­ta­ne­ously.

This goes beyond a sim­ple two-​​way split test. What mar­keters like myself want to know is what com­bi­na­tion of dif­fer­ent vari­ables will have the strongest prob­a­bil­ity of success.

(That’s what “mul­ti­vari­ate” means. Of course, mul­ti­vari­ate, and par­tic­u­larly Taguchi, are a lit­tle more sci­en­tific than this. There are many soft­ware in exis­tence, which are robust and pretty expen­sive since they’re mostly geared for large corporations.)

But one such soft­ware “for the lit­tle guy” is Google Web­site Opti­mizer. It’s free, and you can get it with your Google Ana­lyt­ics account. Their help sec­tion is replete with videos on how to set your tests up, how to test, and how to inter­pret your results.

There are many tuto­ri­als on Google Web­site Opti­mizer, too.

As you know, I’m a fanat­i­cal tester. And I’m already run­ning quite a few split-​​tests with it. I love the results using this new soft­ware and rec­om­mend it. I even share some of my impres­sions and reveal some of my results to my mem­bers at The Copy Doc­tor site.

Now, if you’re not test­ing at all, please don’t fret with all this stuff.

For now, you’re best bet it at least start test­ing.

About the Author

Last 5 Posts By Michel Fortin

• Superior Value Equals Superior Sales
• On Not Playing The Blame Game
• The Future of The Internet is Cloudy
• Carve Your Niche By Dominating One
• Can Copy And Content Commingle?

Other Related Posts

• There's More to Tracking than Conversion (13)
• Transparency Versus Hypocrisy (9)
• The Truth About Mega-Headlines (16)
• The 5 Elements of Defense-Piercing Multimedia (9)
• Splitting Hairs Over Split-Testing? (1)