Website testing and optimization involves the utilization of software to compare the performance of two or more versions of a page or process to determine which version is the most effective in driving desired user behavior.  Tools such as Google Website Optimizer and more robust options geared toward the sophisticated enterprise such as SiteSpect, are available to automate (as much as possible) the process of conducting A/B or multivariate online tests.

Web analytics expert, Eric T. Peterson of Web Analytics Demystified and website optimization software creator, SiteSpect, recently teamed up to release a white paper entitled, “Successful Web Site Testing Practices.”  This must-read document provides a road map for setting up and leading a web testing team within an organization and leveraging it to yield definitive and dramatic results.  Below is our take on some of the highlights of their recommendations.  However, we strongly recommend you read the document in its entirety and take full advantage of case studies, testimonials, and direct examples including a sample test plan approval form (highly useful)!

Best Practice #1: Form a Great Testing Team
Your testing team must have a mandate for improvement.  This means they need to be charged with improving the website’s condition in a measureable way and be able to clearly demonstrate and communicate results to stakeholders.  It’s crucial to include your organization’s top talent in this team; those folks with the clout and track record to spark enthusiasm and buy-in throughout the company.

Eric T. Peterson labels the project manager and the executive sponsor as the two most important roles within the team.  We couldn’t agree more.  Without a carefully managed process and unwavering support from executive management, objectives can be unclear and the powerful force of entropy can take hold quickly.   Of course, without a key executive at the helm of this initiative, team members will find themselves, as Eric puts it, dead in the water.

The author also values the ability of the project manager to think from the perspective of an end user.  After all, it’s the user interface that’s being optimized, and without a firm grasp of user’s needs and the distinct problems or challenges they face when using the site, tests will be misguided and ineffective.

The efforts, updates, successes and even failures of the testing team should be broadcasted throughout the organization.  Allow for a platform for team members to showcase their results.  Ensure this is not merely one-way communication.  Follow the lessons of social media and allow for comments and suggestions in a structured way.  Demonstrating top-level commitment of both time and critical resources communicates the importance of the initiative.  The changes that incrementally occur to the site as a result of testing won’t be alarming to departments.  They’ll appreciate being informed and involved, and will be far more likely to lend their support.

Best Practice #2: Get Your Stakeholders On Board
Like any website redesign, web analytics, or social media endeavor, without management’s support for your testing initiative, you can essentially guarantee failure.

Be sure to work with management from the very start.  Illustrate to them exactly what you wish to achieve and the results you can expect, communicated in bottom-line financial predictions.  Communicate that a data-driven culture is one that strives to make web improvement decisions based on performance and demonstrated merit rather than assumptions.  As a result, an organization is better able to compete in the marketplace.

Best Practice #3: Write a (Formal) Testing Plan
There needs to be a structure through which requests for testing are considered.  Without this, individuals may try to “skip in line” to push their desired updates through based on their own agenda. They need to understand that a prioritized queue exists and suggested tests must be justified by answering the following questions:

1. What is being tested?
2. Why is it being tested?
3. What are the expectations for the test?
4. What are the measures of the success for the test?
5. What are the risks associated with running the test?
6. What internal resources are required to run the test?
7. Who is requesting the test?
8. By when are the results needed?

As mentioned earlier, the white paper authors have included a valuable sample test plan intake form that expands on the above questions.

Best Practice #4: Think about Measurement
Your web testing program should integrate and collaborate closely with your overall web analytics efforts.  This takes extra consideration and possibly technical expertise to ensure data available through analytics tools (such as audience segments) can be applied to tests.  Metrics and KPIs that have been established within a web analytics program should dictate to a large extent which tests should be undertaken.  Make sure one hand is talking to the other.

Also consider how your tests impact more complex measurements such as “return visitation rate” and “lifetime customer value” beyond just clickthrough metrics.

Best Practice #5: Clearly Define “Success” and “Failure”
Success means different things to different stakeholders.  It can range from obvious financial gains, to user engagement, to fewer support calls.  An important point that the author points out is that even a “failed” test yields some measure of success.  That is, you now know more about what doesn’t work thereby narrowing the field of possible creative or interface options getting you closer to the winner.  Don’t view missing the objective as a failure, but rather a learning experience which has actually moved the process forward.  The only real failure is a test that is not properly designed, carefully executed, done within a vacuum, or without executive buy-in.

Best Practice #6: Test Your Test
With this best practice, the author reminds us that while testing is absolutely worthwhile and a potential boon to any company, it’s also “hard.”

Be aware that expectations must be set.  This isn’t going to solve all the issues of the website in a month, quarter, or even year.  This is a long-term shift in the thinking of the company to become data-driven.  Certain technical implementations of tests may be trickier than others and will involve the IT department.  Legacy systems can affect the types of tests that can be conducted.  SEO initiatives can be impacted.  Other departments may push back due to perceived technical or business risks.

Be aware of these challenges, but don’t let them discourage you.  Things of great value don’t come easily, yet are worth the effort.  Set expectations to mitigate disillusionment.

Also, make sure testing periods are long enough to yield a statistically significant sample size.  Isolating factors (individual changes) sufficiently so one can determine which is responsible for better results is crucial as well.

Best Practice #7: Clarify Your Testing Timeline
Testers often forget to take day parts and weekends into account.  To ensure you’re not giving too much weight to lunchtime, early morning, late night, or weekend visitors, allow for enough time to collect a representative sample.  The author recommends a 7+1 strategy so you test over an entire week, but also add one day at the beginning of the test before you actively track results.  This allows the test versions to take hold online across the sample and for any last-minute changes.

Those with more time available can opt for a two week test, in this case a 14+1 timeline.   Good things come to those who wait.

Best Practice #8: Communicate Your Test Results with Actionable Analysis
Some great points here.  As always, it’s best to customize one’s report based on the audience.  When communicating the results to the stakeholders who originally requested the test, go into significant detail.  Reiterate the purpose of the test, the timeline, resources used and extensive information on the results.  If external events occurred during the test such as a major news mention or campaign, illustrate how the results were affected.

The rest of the organization should receive a broad overview.  A big picture, clear summary of the results and the impact on the website and company is sufficient to foster a sense of involvement and ongoing support.  Whenever possible, communicate the financial impact of the test.  As Mr. Peterson puts it, a phrase like, “and we expect this change to result in an estimated $20,000,000 in incremental revenue every year” is hard to ignore.

Also, check out the white paper to see a ready-to-use email example of how to broadcast test results to the organization.

In addition, in-person presentations are always best.  Follow these meetings up with postings to your communication platform be it a blog, wiki, online project management tool, or even Google Wave.  Allow enthusiasm for the results to go viral.

Above all, include actionable recommendations along with your results.  Clearly state what next action steps should be taken to capitalize on the success of the test.  Use this opportunity to suggest additional tests to keep the optimization ball rolling.

Best Practice #9: Test Different Audience Segments
The most revealing tests will be those that utilize targeted audience segmentation.  Knowing that a certain change increased conversions by 5% for all visitors is somewhat helpful, but knowing that same change resulted in a 20% increase for a key target audience is far more valuable.  Next action steps are more obvious in light of this information and those steps are more likely to have greater impact.

Directly reference audience segments when presenting test results to stakeholders to better grab their attention.

Best Practice #10: Mine for Deeper Opportunities
Once the testing program is up and running, and the team has a few wins under its belt, in-depth data analysis and statistical modeling is the next level to strive for.  Companies that are already utilizing robust applications such as SAS, or SPSS have an advantage, but this is not an absolute requirement.

Analysis and comparison of offline data or qualitative voice-of-customer data alongside test results can yield insights that are not immediately obvious through the testing or web analytics tools alone.  The author points out that a significant competitive advantage will be had by those companies who truly leverage their online data.

Your Experience?
We’d love to hear your thoughts on web testing and optimization.  Have you recently formed a team or are you considering it?  What methods and/or tools have brought you success?  What real world challenges has your organization overcome (or is still struggling with)?

Interactive Industry News & Events, Internet Marketing, User Interface Design, Web Analytics, Web Project Management, Web Strategy Consulting Google Website Optimizer, User Interface Design, Web Analytics, Web Project Management, Website Testing