Multivariate testing seems to be an easy enough process to implement: have a question about a website or online newsletter’s design and its appeal to customers? Test it!
Indeed some best practices in this space would seem to be almost counter-intuitive.
Consider the following:
1. For best results use both quantitative and qualitative measurements. It is quite common for seemingly dramatic design changes to have no significant impact when examined using simple measures such as click-through and conversion rate. That is why an increasing number of companies use more complex measures such as “return visitation rate” and “lifetime customer value” – as well as qualitative measurement systems to develop a more holistic view of the test’s impact.
“For example, Ask.com examines their testing efforts in both quantitative and qualitative terms, using their Voice of Customer system to ask test participants questions about ‘likelihood to return,’ ‘likelihood to recommend,’ and ‘overall satisfaction,'” according to the SiteSpect paper.
Instead of high-end design skills, see if the manager can think from the perspective of an end user. According to John Stansbury, Director of Analytics and Testing at CreditCards.com, the two most important requirements for a testing project manager are the ability to tell a story and the ability to work closely with other people. “Testing is really a sales job,” says Stansbury.
3. Sometimes more chefs – instead of more helpers – are better.
Buy-in from top executives will be important – so important that it is worth the risk of getting sidetracked by pet projects that may be driven more by seniority than need or opportunity. To get their buy-in and get the necessary work done as well, work with these stakeholders from the beginning and directly solicit their feedback, suggestions, and ideas – even if the result are a few less-than-perfectly conceived project.
The reality is that making changes to the site is as much political as tactical. Failing to consider the politics of the situation and attempting an end-run – or “embarrassing the boss” as one consultant puts it – does little to promote the value of testing.
In fact establishing a multivariate testing steering committee works at some companies for this reason – plus makes sure the trains run on time as it considers what will be a wave of testing possibilities. In other words, it helps ensure that there are the right resources to test both the simple things and the big projects.
4. There is no straightforward definition of success. For some companies, “success” is a dramatic increase in a revenue-based metric, knowing that most senior stakeholders will respond to incremental revenue.
For others, “success” is a small increase in key visitor engagement metrics, knowing that a series of small gains eventually adds up. For still others, “success” is a reduction in the number of problems present throughout the site, knowing that reducing barriers improves usability.
And for those companies with an increasingly dated site, “success” is simply being able to deploy a new look without negatively impacting existing key performance indicators.