Common Mistakes in A/B Testing

[Rakhiraqsdiwseo]

At Convert.2014, Alexander Oleshchuk (usability analyst and designer at e96.ru) gave an interesting presentation on A/B testing.

He talked about the main mistakes and simple ways to start A/B testing.


Content
Common Mistakes in A/B Testing
1. Inflated expectations (hello, Amazon!)
2. Heterogeneous sample
3. Insufficient sampling
4. Traffic heterogeneity
5. The hypothesis is not defined
A/B testing services
I have an impulse to "freak out and redo everything" every now and then. This is not even close to analytics, and you shouldn't do it. Everything is clear here But there are more subtle mistakes.

1. Inflated expectations (hello, Amazon!)
Almost everyone who has heard of A/B testing falls under the spell of the “fairy tale” about the “long and short page on Amazon.” In less sophisticated cases, about repainting the “order” button and wildly increasing conversion.

Be prepared for modest results. Also, the absence of any difference is also a result.

2. Heterogeneous sample
Conversion with SEO ≠ conversion with chinese overseas british database advertising. Need 1 type of traffic for testing.

3. Insufficient sampling
Don't do that (about A/B testing)

The report contains a formula for calculating the minimum sample. Use it.

4. Traffic heterogeneity
Conversion from SEO may differ from conversion from contextual advertising - you need 1 type of traffic for testing.

Also, you should only test on new visitors. This is because old customers already have some experience (positive or not), while you need a “clean slate” view.

5. The hypothesis is not defined
Alexander Oleshchuk seems to be telling us: who tests like that?!
Alexander Oleshchuk seems to be telling us: without a hypothesis, who tests like that?!
The customer usually says: "The main thing is to make more money." This is the right direction, and at the same time it needs to be specified.

Example hypothesis: "additional products do not increase conversion."

This hypothesis is easily verified:

you know what you're going to enter,