Select Page

There are a LOT of people out there who are generating some impressive results with simple split-testing. Why not? There are tons of great platforms to make the process of testing relatively easy.

Imagine this, you split test your existing homepage call-to-action with a new one. A few days later, you see that the results are showing the new variation is handily beating the original page. You declare the obvious leader as the winner and make the change permanent. How could this possibly be anything but good?

Following are a few scenarios where this might be a bad move:

  1. While the change made more people take action, it also drove more people through your marketing funnel that do not convert and actually cause your bottom funnel conversions to decrease.
  2. Declaring a winner before you truly have enough data can be troublesome even if your testing tool declares a winner. With just about every test I’ve done, the first few days are wonky at best. The numbers are all over the place and not evenly remotely indicative of the results you’d see after things normalize.
  3. If thorough testing of the pages wasn’t done when the testing was turned on, your winner might only be because there was a coding error in the original page.

There are many other possible ways that the new page might be costing you money in the long term, but those are three that stand out before all the others.

Here some tips on how to split test to avoid these mistakes:

  1. Test in full weeks. Many people have usage habits that are weekly. You want to get an equal number of each day of the week in your test. To do this, determine how much traffic you need for statistical significant data. At least 1000 unique views per variation is a good place to start. (Sidenote: Some of the well-known testing tools, even the ones we use, have the default minimum sample size as low as 100) Then, using your website analytics, determine how many days you need to get that sample size. Round that number up so that the test ends on the same day of the week that it will start. You can also give it a few extra days and remove the first two or three days to clear out that test start wonkiness.
  2. Follow your customers through your entire marketing funnel to determine if the change you made is truly having a positive effect on your KPI’s. Every good CRM has ways of doing this. If your’s doesn’t, you can try using hidden form fields with custom UTM tracking links. Raven Tools has a good URL builder for Google Analytics.
  3. Make sure to test and re-test the pages when you make the split test live. Test using a Chrome Incognito window and/or test from multiple IP addresses and browsers so that you discover any conversion killing code errors. Also, if you are testing mobile traffic in this test, make sure to adequately test across various devices including tablets.