My guess is they have been doing automatic experiments on our traffic already - given the number of times I see ad colours that don't relate to my setup.

I'm glad that they are offering some data about the results now.

I turned it on as soon as I saw it, maybe 2 days ago. Will be interesting to see because so far their automated suggestions aren't always right. And, yeah I think we all know they did and do experiment with our traffic, so why not let us in on the details of the experiment? I see this as a good thing.

I am always sceptical of Google and Microsoft suggestions.

I wonder why that is so?

I can't actually turn it on - experiments worked for a day or so and let me close down all my previous experiments - then when I logged in the next day it just came up with "Sorry we are experiencing technical difficulties" or something of that ilk (as it had for me for a couple of months prior to this) and wouldn't let me turn on the auto-experiments

Anyone know exactly how this is supposed to work? Will it try different sizes? Different positions? Will it affect the layout of my site?

No. That may come at some point, but not now.

So far if they are running automatic experiments, I'm not seeing anything showing up in the list of experiments.

I take that back. It turns out I have 9 automatic experiments already. They put them at the bottom of the experiments page so I missed them. The first experiment that's already complete does not tell me what it tested. It just says "Original" and "Variation". The experiments that are in progress at least tell me what it's testing.

The first experiment that's already complete does not tell me what it tested. It just says "Original" and "Variation".

Ah yes, the old "double secret probation" experiment...

I sent them feedback pointing out the obvious problem with that. Hopefully they'll be a bit more forthcoming with the details...

I started some tests with this last week, and so far have been disappointed. Even after 20k views in one test, it still shows 50% conclusive (ie, as certain as flipping a coin), and it only allowed me to change the font style/colors of an ad or whether I show text ads vs image ads. Not a lot of options here unfortunately :(

Anyone have a rough idea when new Experiments change from "Collecting data for analysis. Please check again later." to providing some actual feedback?

Apparently not til a clear winner with a high confidence level emerges. If your test is showing 50/50 even after a gazillion impressions, then there's no clear winner so it will still say collecting data - until and if it ever does.