Hi,

How much can one rely on Experiments A/B? I know it's the only tool, but I think that there are many elements that could change/tilt the earnings balance: luck, budget of advertisers that could vary from one month to another, the time that one would run the experiment if it's one week I guess it's not representative, if it's one month is it then enough?

As far as I see it, trying to understand why a day/week/month earnings were higher than another month, except for the number of visitors (e.g. in a month more visitors than in another), it's almost impossible to draw any conclusion.
It's like the stock exchange, many think that they understand the reason but if there is an unfortunate coincidence that 3-4 big shareholders would sell many shares, the price would still drop even though other similar shares might increase...

What I'm trying to say is that there are so many elements including the luck/random that it would be difficult to infer something from the A/B experiments.

I'm not having that many visitors, quite few, around 1000/day, I guess for higher number of visitors, the chance of having a better distribution of users, and a better statistical coverage (e.g. richer visitors, middle class, older, younger, different locations....) is higher and there will always be a mix of users. I guess that the percentage of visitors would not be the same.

Sorry for the long post, my only question is how many days/weeks/months would you run the A/B experiments to make it statistically reliable? I guess that the number of days/weeks/months should be also extended if there are less visitors to allow for higher chance of having a more diverse visitor base.

Many thanks for your replies