Splitting Hairs Over Split-Testing?

After the launch of our new auto­mated split-​​testing soft­ware, con­tro­versy soon followed.

Most of it was caused by the math behind it (which is com­plex and under­stand­ably hard to grasp for most peo­ple), or by com­peti­tors spread­ing misinformation.

So peo­ple were either con­fused by it or mis­un­der­stood it.

For exam­ple, some peo­ple posted in pop­u­lar forums that my copy for Muti​Track​Gen​er​a​tor​.com is mis­lead­ing because it’s not pure Taguchi.

(Well, tech­ni­cally, they’re right.)

Here’s why.

I can­not speak for Marc or Armand. But since I’ve had lengthy dis­cus­sions with both, since the soft­ware was based on what I wanted (and I wanted Taguchi), and since I wrote the copy, I was com­pelled to explain a lit­tle further.

First of all, Mul­ti­Track Gen­er­a­tor is based on Taguchi and does use Orthag­o­nal Arrays (i.e., the math that makes Taguchi work) in its algo­rithm. (A bet­ter term may be “inspired” by Taguchi, because it is indeed not entirely pure. But I will return to this.)

There is a rea­son why I used the term “bet­ter than Taguchi” in the copy. We don’t just throw around the word “Taguchi” for the mere sake of rid­ing the buzz bandwagon.

(Believe me, it took over a year for Marc Quar­les, the pro­gram­mer, to learn, under­stand and pro­gram the math cor­rectly into the soft­ware, as Taguchi is indeed very com­plex and advanced.)

Let’s clear up some mis­con­cep­tions, shall we?

There’s a huge dif­fer­ence between “multi-​​variable” and “multi-​​variate” (or “multi-​​variant”). The for­mer is sequen­tial. It’s run­ning mul­ti­ple split-​​tests simul­ta­ne­ously — yet inde­pen­dently, because com­bi­na­tions are not observed.

Sure, it’s a step up from tra­di­tional A/​B split-​​testing. But each test still requires a cer­tain result to deter­mine its accu­racy and importance.

This is time-​​consuming.

What I pre­fer, and what MTG does, is the lat­ter: multi-​​variant (or multi-​​variate) test­ing. And NOT multi-​​variable.

Multi-​​variate is either Taguchi-​​based or Taguchi-​​inspired. I say “inspired” because, while the con­cept is fun­da­men­tally the same as Taguchi, the appli­ca­tion is dif­fer­ent as it has been cus­tomized for the web.

Here’s why.

Mutlti-​​variate (or Taguchi-​​based) split-​​testing, in essence, uses Orthag­o­nal Arrays to closely pre­dict the best pos­si­ble com­bi­na­tion of vari­ables with a smaller num­ber of actions/​results.

(In the case of a web­site copy­writer, for exam­ple, it may be the best pos­si­ble com­bi­na­tion of, say, head­line, back­ground color, open­ing para­graph, bul­let list, offer, price, guar­an­tee, call to action, PSes at the end, etc, with the least num­ber of vis­i­tors and the least num­ber of sales each vari­able generates.)

Taguchi thus allows you to make a judg­ment call based on far less results/​actions by sta­tis­ti­cally “guess­ing” which com­bi­na­tion will pro­vide the best con­ver­sion with only a hand­ful of tests.

But Taguchi, by itself, has its flaws — par­tic­u­larly when applied to the Internet.

I’m not a math­e­mati­cian, and I’m sure Marc can explain this bet­ter than I can. But here’s my “shot­gun” explanation:

(By the way, this does not reflect MTG, or the views of Marc or Armand. They are solely mine and my opin­ions alone. Note that I have NO inter­est in this soft­ware. In fact, I didn’t even get paid for writ­ing the copy. I did it for a friend.)

Taguchi is pow­er­ful math designed for the car man­u­fac­tur­ing indus­try. Nor­mal Taguchi meth­ods only apply the cal­cu­la­tions after the test­ing is com­pleted. Which is really the only way you can do it — because you must pro­duce an actual car that needs to be tested only once the prod­uct is phys­i­cally devised.

(You can’t test “an idea” of a car.)

Taguchi thus allows you, using its com­plex math, to sta­tis­ti­cally deter­mine the prob­a­bil­ity of suc­cess with a com­bi­na­tion of car com­po­nents using fewer test mod­els (or, in this case, cars.)

If you’re using 12 dif­fer­ent car com­po­nents and you want to deter­mine which com­bi­na­tion of these com­po­nents work best together, then you nor­mally would have to pro­duce, say, 12 x 12 (i.e., 144) test cars to actu­ally test which com­bi­na­tion is the best.

(Again, I’m no math­e­mati­cian. But I think you under­stand my point.)

Dur­ing WWII, Japan’s resources were scarce. So Genichi Taguchi came up with a way to save money by cre­at­ing less tests mod­els (i.e., less cars) by deter­min­ing which out­side influ­ences exist (called “noise”), and to which degree they affect the end result.

That way, using Taguchi we can deter­mine, in advance, which influ­ences could be iso­lated and elim­i­nated, and which ones would have an impact, and above all, to what degree.

Con­se­quently, you can esti­mate the best com­bi­na­tion of car com­po­nents with far fewer cars, thereby sav­ing you loads of time and man­u­fac­tur­ing costs. Using my ear­lier exam­ple, rather than cre­at­ing 144 cars, you only need to cre­ate, say, 10 or 8 or whatever.

And for the same rea­son, Taguchi works well in the case of direct mail­ings because the results are only deter­mined once the mail­ings have been con­ducted and the test performed.

Sim­i­lar to man­u­fac­tur­ing, you can do a few tests, crunch your data, “Tagu­chinize” them and make an esti­ma­tion on the best pos­si­ble com­bi­na­tion with a greater degree of accuracy.

But it’s not so with online salesletters.

The Inter­net is dynamic. Sure, Taguchi is great for ads, ban­ners and small web pages. The dif­fi­culty lies in deter­min­ing Orthag­o­nal arrays for unknown num­bers of vari­ables per element.

And with online saleslet­ters, for instance, the poten­tial per­mu­ta­tions are vastly greater than, say, a sim­ple Google AdWords ad or a small web page.

With nor­mal Taguchi, there is a dis­tinct cycle of create-​​test-​​evaluate.

You have to do it this way because you must first con­duct a lim­ited num­ber of tests and run your test results through Taguchi ONLY once those results are achieved.

With the Inter­net and espe­cially with long copy saleslet­ters, you have the abil­ity to opti­mize your ad dynam­i­cally. That’s not all. You also have a far greater num­ber of poten­tial combinations.

And more impor­tant, out­side influ­ences can also change quite rapidly — some­times dra­mat­i­cally.

That’s why the prob­lem with cur­rent Taguchi-​​based soft­ware in their cur­rent state is that it’s not flex­i­ble enough to work with long copy saleslet­ters as it can only com­pute a spe­cific num­ber of tests, and do so only after the tests are con­ducted and results tabulated.

Once this step is done, it gives you an impor­tance level attrib­uted with each fac­tor so you can make a judg­ment call on the depend­abil­ity and accu­racy of the result­ing favored fac­tors. Then you move on to the next set of tests, if you wish. And so on.

So it’s quasi-​​sequential to a degree — not because of Taguchi but because of the rigor, demands, speed and com­plex­ity of long copy salesletters.

That’s why, even though Taguchi reduces the num­ber of tests required and the time it takes to reach a cer­tain result, cur­rent Taguchi-​​based soft­ware is sim­ply not enough.

What MTG can do is deter­mine the impor­tance that the Taguchi func­tion gives you while test­ing is still going on, and apply it to the ele­ments as they are still being tested.

So for exam­ple, if the head­line is way more impor­tant than the back­ground color, then the aim is to favor the bet­ter head­line for the remain­der of the test­ing because it is the most impor­tant element.

If the sys­tem can esti­mate which one is best in com­bi­na­tion with all the other vari­ables and poten­tial com­bi­na­tions, it should use that exclu­sively while the remain­der of ele­ments get tested.

So ulti­mately, the over­all con­ver­sion ratio may be improved dur­ing the test­ing process. Not after.

That’s the beauty and the chal­lenge of the Inter­net, and its abil­ity (and also, I might add, the necess­esity) to con­duct real-​​time, dynamic test­ing — which Taguchi is not really meant to work with.

That’s why I said it’s “bet­ter than Taguchi.”

Or bet­ter said, it’s bet­ter than the way Taguchi is cur­rently being applied. Because nor­mal Taguchi is sta­tic and lim­it­ing. It’s NOT dynamic.

It was never meant to be.

About the Author

Last 5 Posts By Michel Fortin

• How to Capture and Captivate Attention
• The Real Problem With The Flaw of Attraction
• Two Shining Stars in an Age of Darkness
• Changes, Relaunches, and Extreme Makeovers
• Superior Value Equals Superior Sales

Other Related Posts

• There's More to Tracking than Conversion
• The Real Sinister Side of Forced Continuity
• Secrets of a 10% Conversion Rate
• Put Your Copy To The Test
• Computer Geek Discovers Speed Copy Secrets?