Comments on: The bad science of A/B and multivariate testing for e-commerce http://marketplanb.com/blog/2007/12/19/the-bad-science-of-ab-and-multivariate-testing-for-e-commerce/ Strategies and tactics for online marketing and ecommerce Wed, 26 Mar 2008 19:37:21 -0700 http://wordpress.org/?v=2.9.1 hourly 1 By: Multivariate Testing - Overdoing It? | Data SystemsPlus http://marketplanb.com/blog/2007/12/19/the-bad-science-of-ab-and-multivariate-testing-for-e-commerce/comment-page-1/#comment-1123 Multivariate Testing - Overdoing It? | Data SystemsPlus Mon, 28 Jan 2008 19:32:47 +0000 http://marketplanb.com/blog/2007/12/19/the-bad-science-of-ab-and-multivariate-testing-for-e-commerce/#comment-1123 [...] Why? The first is simple math. Tom Lindmeier explains it better than I could - but the bottom line, without big, stable traffic numbers and a steady, measured conversion rate, your multivariate testing results are not much better than choosing a ‘winner’ sales page at random. Any freshman statistics student learns that for a statistical observation to be reliable, it needs to be derived from a sample size large enough to ensure reliability. If you are launching a new product or site with zero traffic to start - you are making a mistake if you are making copy writing decisions based on statistics based on just a few hundred visitors and a handful of sales. Keep in mind also, the more variables you test, the more observations you will need for a valid test. [...] [...] Why? The first is simple math. Tom Lindmeier explains it better than I could – but the bottom line, without big, stable traffic numbers and a steady, measured conversion rate, your multivariate testing results are not much better than choosing a ‘winner’ sales page at random. Any freshman statistics student learns that for a statistical observation to be reliable, it needs to be derived from a sample size large enough to ensure reliability. If you are launching a new product or site with zero traffic to start – you are making a mistake if you are making copy writing decisions based on statistics based on just a few hundred visitors and a handful of sales. Keep in mind also, the more variables you test, the more observations you will need for a valid test. [...]

]]> By: Tom Lindmeier http://marketplanb.com/blog/2007/12/19/the-bad-science-of-ab-and-multivariate-testing-for-e-commerce/comment-page-1/#comment-779 Tom Lindmeier Sat, 22 Dec 2007 15:23:42 +0000 http://marketplanb.com/blog/2007/12/19/the-bad-science-of-ab-and-multivariate-testing-for-e-commerce/#comment-779 Billy, I agree with you for the most part. I recognize the need to be careful about throwing numbers around unless we state the exact nature of the test. I'm not saying that the Google Optimizer is lacking. (The reference to 1M page views is on page 14 of the <a title="Demo" href="http://services.google.com/websiteoptimizer/">Overview Demo</a>). But I am saying that small and medium sized businesses need to moderate expectations of the number of tests they can run when they have moderate traffic. They are faced with running one test at a time over longer periods. Here's an example: Lets say you're running a home page offer test and a shopping cart test and both are multivariate with 4 treatments each. Even if you assign the lions share of page views to the control group on the home page test, the factor for your cart test is diluted to a point that you cannot run the cart test and possibly not even the home page test unless it is A/B. Overall conversion rates for most e-commerce businesses fall into the 2% to 4% range and that is how I came up with the 20-25M sample size. Billy,
I agree with you for the most part. I recognize the need to be careful about throwing numbers around unless we state the exact nature of the test. I’m not saying that the Google Optimizer is lacking. (The reference to 1M page views is on page 14 of the Overview Demo). But I am saying that small and medium sized businesses need to moderate expectations of the number of tests they can run when they have moderate traffic. They are faced with running one test at a time over longer periods.
Here’s an example: Lets say you’re running a home page offer test and a shopping cart test and both are multivariate with 4 treatments each. Even if you assign the lions share of page views to the control group on the home page test, the factor for your cart test is diluted to a point that you cannot run the cart test and possibly not even the home page test unless it is A/B.
Overall conversion rates for most e-commerce businesses fall into the 2% to 4% range and that is how I came up with the 20-25M sample size.

]]> By: Billy Shih http://marketplanb.com/blog/2007/12/19/the-bad-science-of-ab-and-multivariate-testing-for-e-commerce/comment-page-1/#comment-774 Billy Shih Fri, 21 Dec 2007 19:39:08 +0000 http://marketplanb.com/blog/2007/12/19/the-bad-science-of-ab-and-multivariate-testing-for-e-commerce/#comment-774 Disclaimer: I am an optimization analyst at Widemile and we specialize in multivariate and split testing. I can see where you are coming from with your issues, but I think there are a lot of misconceptions and mistakes going on with multivariate and a/b testing since it is a very nascent industry in the online world. I agree that sample sizes are important and that it is unrealistic for every business to be doing large multivariate tests or even split tests, but there are some problems with some of the things you suggest. Any test that lasts 6 months to a year is probably invalid in itself because that is too long of a time period. If you sell Ski's, you probably get different types of traffic in the Winter than in Spring, so optimizing your site over that long of a period will skew your results in different directions. The longer your test is, the more noise you introduce to your results, which at a certain point makes your results not statistically relevant. Even for non-seasonal products, assuming that 6 months to 1 year has no significant traffic changes is probably unreasonable since most companies are doing PPC and other advertising changes along with SEO over that time. Also, multivariate testing requires a certain amount of conversion traffic only 99.9% of the time. This is simply because conversions are much harder to get, so the sample size of page views is almost never a problem. Also, you can not say that you took into account for conversion traffic with such a narrow number from 20-25M. My company has tested sites with conversion rates as low as .1% all the way up to 30%, which is a huge difference in the amount of traffic required. You have to remember that some multivariate tests take longer than others also. A very basic one could take a site one week, while a very complicated one could take the same site 4 weeks or more. While we rarely work with small businesses, many of our clients are medium sized. Not every business fits but if they do, its because of their conversion traffic. We have not had problems optimizing their pages using split and multivariate testing and we get statistically relevant results within a month or less typically. Google's tool, while not perfect, does a great job at driving real results. I'm not sure where you got the 1M page views a week number from, nor the context of it, but from what I've seen, their calculations of how long it takes to run a statistically significant test have been accurate. Since their tool is free, and only helps to boost their AdWords revenue, Google has no incentive to give people a tool that gives them bad results. Also, I can't speak for other companies, but it is in our best interest to create long term conversion lifts for our clients. Many times I have pushed clients tests to run a test longer simply because we need to get solid results. I think you should give Google Optimizer another try, if you haven't already and see if you still think the same way. As long as you follow their guidelines of not testing too many things in regards to your traffic, it works. Disclaimer: I am an optimization analyst at Widemile and we specialize in multivariate and split testing.

I can see where you are coming from with your issues, but I think there are a lot of misconceptions and mistakes going on with multivariate and a/b testing since it is a very nascent industry in the online world. I agree that sample sizes are important and that it is unrealistic for every business to be doing large multivariate tests or even split tests, but there are some problems with some of the things you suggest.

Any test that lasts 6 months to a year is probably invalid in itself because that is too long of a time period. If you sell Ski’s, you probably get different types of traffic in the Winter than in Spring, so optimizing your site over that long of a period will skew your results in different directions. The longer your test is, the more noise you introduce to your results, which at a certain point makes your results not statistically relevant. Even for non-seasonal products, assuming that 6 months to 1 year has no significant traffic changes is probably unreasonable since most companies are doing PPC and other advertising changes along with SEO over that time.

Also, multivariate testing requires a certain amount of conversion traffic only 99.9% of the time. This is simply because conversions are much harder to get, so the sample size of page views is almost never a problem. Also, you can not say that you took into account for conversion traffic with such a narrow number from 20-25M. My company has tested sites with conversion rates as low as .1% all the way up to 30%, which is a huge difference in the amount of traffic required. You have to remember that some multivariate tests take longer than others also. A very basic one could take a site one week, while a very complicated one could take the same site 4 weeks or more.

While we rarely work with small businesses, many of our clients are medium sized. Not every business fits but if they do, its because of their conversion traffic. We have not had problems optimizing their pages using split and multivariate testing and we get statistically relevant results within a month or less typically.

Google’s tool, while not perfect, does a great job at driving real results. I’m not sure where you got the 1M page views a week number from, nor the context of it, but from what I’ve seen, their calculations of how long it takes to run a statistically significant test have been accurate. Since their tool is free, and only helps to boost their AdWords revenue, Google has no incentive to give people a tool that gives them bad results.

Also, I can’t speak for other companies, but it is in our best interest to create long term conversion lifts for our clients. Many times I have pushed clients tests to run a test longer simply because we need to get solid results. I think you should give Google Optimizer another try, if you haven’t already and see if you still think the same way. As long as you follow their guidelines of not testing too many things in regards to your traffic, it works.

]]> By: Tom Lindmeier http://marketplanb.com/blog/2007/12/19/the-bad-science-of-ab-and-multivariate-testing-for-e-commerce/comment-page-1/#comment-763 Tom Lindmeier Thu, 20 Dec 2007 19:51:58 +0000 http://marketplanb.com/blog/2007/12/19/the-bad-science-of-ab-and-multivariate-testing-for-e-commerce/#comment-763 Craig, Thanks for the visit. I assume that you're referring to site optimization relating to site design, cart functionality, etc. Not landing page testing. Actually I'm saying that tests could take up to six months or a year. The 20-25M number takes conversion into account so you are not dealing with an inordinately small number of orders. The lower your expected conversion rate, the higher your test count should be. Visits rather than page views would be your criteria for test sample size (depending on your test). Page views or click-thrus would be the basis for landing page tests. Yes, an A/B test is better but may require a second or third round of testing so it could take more than a year to test one hypothesis. Nobody wants to hear that because we all want to move swiftly when out-flanking the competition. Craig, Thanks for the visit.
I assume that you’re referring to site optimization relating to site design, cart functionality, etc. Not landing page testing.
Actually I’m saying that tests could take up to six months or a year. The 20-25M number takes conversion into account so you are not dealing with an inordinately small number of orders. The lower your expected conversion rate, the higher your test count should be. Visits rather than page views would be your criteria for test sample size (depending on your test). Page views or click-thrus would be the basis for landing page tests.
Yes, an A/B test is better but may require a second or third round of testing so it could take more than a year to test one hypothesis. Nobody wants to hear that because we all want to move swiftly when out-flanking the competition.

]]> By: Craig http://marketplanb.com/blog/2007/12/19/the-bad-science-of-ab-and-multivariate-testing-for-e-commerce/comment-page-1/#comment-762 Craig Thu, 20 Dec 2007 16:01:08 +0000 http://marketplanb.com/blog/2007/12/19/the-bad-science-of-ab-and-multivariate-testing-for-e-commerce/#comment-762 I think you're assuming tests only take a week. Most experts recommend tests over a month. Also, doesn't significance depend on the conversion rate, relative difference in treatments, and number of treatments as much as it does on traffic? Lastly, if you don't have a lot of traffic, isn't a simple AB test still fine? I think you’re assuming tests only take a week. Most experts recommend tests over a month. Also, doesn’t significance depend on the conversion rate, relative difference in treatments, and number of treatments as much as it does on traffic? Lastly, if you don’t have a lot of traffic, isn’t a simple AB test still fine?

]]>