There are 11 classic mistakes that can actually cause a company to lose customers though standard A/B testing. What can your company do to stop them? In an article for CIO.com, Jennifer Lonoff Schiff shows how to get the data you want without alienating anyone and without wasting effort.
- Testing too many elements at once
- Testing the obvious
- Testing something insignificant
- Testing something undeliverable
- Testing the wrong thing or making false assumptions
- Running an A/B test at different times
- Using flawed landing pages
- Using too small a sample size
- Not using segmentation
- Ending tests prematurely
- Reporting results before the test is complete
Perhaps the most obvious mistake is that of (1) having too many test variables. CMO Corinne Sklar cites the example of using split subject lines in an email body. In this case, how would it be possible to determine which factor was generating leads?
Another no-brainer is (2) to avoid testing for content solutions that have already been found, such as “Dear Customer” versus “Dear First Name.” Everyone knows that personalization wins the day. Along those same lines it (3) makes no sense to test something that’s totally insignificant from the customer’s perspective. In other words, “Test apples against oranges first,” says Justin Talerico, founder & CEO, Ion Interactive…”
What’s worse than testing something insignificant? Answer – (4) testing something significant that can’t be delivered. The insights may be eye opening, but your time is wasted if you can’t deliver on those items. But what if you find yourself testing (5) the wrong thing? As is often the case when managers make false assumptions, things that customers never ask for may get tested. Always question the A/B’s underpinning logic.
And (6) always, always, run your A and your B at the same time – otherwise you’ve defeated the purpose of the test by ruining the common variables. By the very same token, each landing page must (7) be identical. In any rigorous testing environment, there needs to be ample consistency in addition to (8) an appropriate testing size. An additional recommendation is to (9) use market segmentation when analyzing your data. Establish a sound strategy ahead of testing, with proper cohorts targeted. One can also segment by counting the number of times each user has committed to X action.
It’s important not to (10) end tests too early, as this will compromise the integrity of the test. By that logic, don’t get (11) too excited or impatient with results. In due time, the test will end and the report can be written.
Read the original article at: http://www.cio.com/article/2906043/marketing/11-a-b-testing-mistakes-and-how-to-avoid-making-them.html