Here's something that grabbed my attention:

The team was struck by how few of the outsiders noticed that the search results were changing rapidly below the search bar, Boyd said... less than half of the outsiders noticed Google Instant during the first series of tasks.

That's something that I also noticed talking to other people who are not SEOs. Seems to me that we SEOs need to get out more ;)

It is because most people watch keyboard and not screen when they type...

WOW, only 160 and 80 of them were Googlers. That doesn't seem like a very large number for an accurate test for such a big change.

WOW, only 160 and 80 of them were Googlers. That doesn't seem like a very large number for an accurate test for such a big change.

I've seen city council races where more people were polled. With billions of dollars at stake, you'd think they'd use a larger focus group.

The testing (it's not just a poll) was done over a 6 month period turned out extremely strong. I have no doubt that the data were statistically relevant. After all Google does employ many PhDs in statistics.

From what I've seen so far in the server logs since Instant rolled out, predictions that Google had finally jumped the shark did not come true. In fact, the practical effect seems to be a big yawn.

WOW, only 160 and 80 of them were Googlers. That doesn't seem like a very large number for an accurate test for such a big change.

I used to conduct market research, this is a very big sample for this type of test. This is qualitative research, not quantitiative (such as the poll example). You're not trying to predict the next President or anything...

It's still ridiculous that a company of this size tests 160 people. When I first read this I thought it was a joke.

80 were googlers which means they wouldn't rock the boat anyway. I'd work for free and in three months could work with the spam team to level the field...haha.

After the 160 person development test, Instant was also tested live [ousbey.com] for a subset of users before it was finally launched.

Google reminds me of the character played by alec Baldwin on 30 rock. In one episode he and his team work hard to develop a new product to save ge, a new toaster. After toiling with it for days his team says it looks great, when he sees it he remarks "congratulations we've invented the Pontiac aztek" in reference to one of the ugliest consumer products ever witnessed.

Google 2010. Their next update should be called aztek in honor of a year spent bringing features forward that nobody wanted. Instead of giving us the homer Simpson Canyonero with every feature a couple of over caffeinated phds thought we'd like how about just filtering the spam. And by that I mean hire a couple of spammers to help root out the spam. They could set up a chat deal and probably parse thousands of legitimate spam reports in a few weeks that would give them more useful data on spam trends the all those useless forms

I'm nearly going to do a U-turn on my comments. I've just had a client emailing me with screen shots asking why they can't see their ads. All the screen shots indicated they were using Google instant, the tell tale arrow on the top result (it's sponsored too) and the "instant is on". The first emails go back a few weeks, but I hadn't really paid attention as his ads weren't on the page due to budget constraints.

I spoke to him and asked him what he thought of Instant. He had never heard of it and didn't even notice it was happening at all until I talked him through it. He did think it was cool though.

I think I'll get a few more opinions from people who don't spend 4+ hours a day on Google

The team was struck by how few of the outsiders noticed that the search results were changing rapidly below the search bar..

Wow...What does he mean by that? they decided to push in this change because very few outsiders notice the dance! If they don't even notice the dance, how will it benefit the "Users"?

I may be wrong but from that observation, it looks like the primary objective of Google Instant is something else! It looks like they decided to push in the change to meet their unstated objective, as a fewer percentage of the ordinary users are going to notice the "Instant dance"!

One thing I have noticed is you get a nice little blue arrow pointing to the top AdWords ad (as long as there are ads above the organics) after you have entered your query and hit enter or search.

This draws extra attention to the ad & if they can increase the CTR of the No 1 ad by a few percent they are going to make a killing in extra click income.

It also takes you to that advertisers page if you click enter after your initial search.

If you're ad is is no 1 all the time, have you noticed an increased CTR recently? This could explain it.

80 were googlers which means they wouldn't rock the boat anyway.

Right, Googlers wouldn't say anything because they're afraid of... What?

Oh yeah, you're going to recruit people in the company and if they don't like it, then you're going to fire them. Yeah, right. That what we did when I worked at Conspiracy Theory Inc.

Not my point at all. There are 23,000 google employees and 200 million web users in this country alone. A representative sample of john public isnt going to come from a set of users so heavily skewed towards the company.

This has been a bumpy year for google results. There will be a lag in any significant change towards bing. If the changes this year at google are a prelude to better things ahead bings share will fall again. If google pulls a windows millenium and makes a change to the algo that sticks around long enough people start using bing again....

Live doesn't do much for me but it seems faster than a few weeks ago and when it's slow it tells me the connection is bad versus lagging like it did. I'd much rather see work done on the layout versus this stuff.

If the changes this year at google are a prelude to better things ahead bings share will fall again.

That's your opinion. Your assuming that Bing users have a reason to leave Bing and try Google and as a Bing user, Google has lost me forever and as long as they still return descent results, people will use them and have no reason to leave.

I find that many webmasters (not average users) who claim Bing is horrible and returns mostly spam are usually webmasters who can't get their sites to rank on Bing.

One thing I have noticed is you get a nice little blue arrow pointing to the top AdWords ad (as long as there are ads above the organics) after you have entered your query and hit enter or search.

If you arent aware, if you then use your direction keys this arrow moves to select a result then hit enter to navigate to it - anothe gimmic which even some people in the industry dont even know about! There is a topic on it I think.

Google instant is a complete gimmic, for many of the reasons already mentioned. It would of course get a thumbs up for cleverness which could be why it got through the panel of testers but the practical applications are slim to none to the wider searching audience.

That they only tested with 80 outside viewers given the set of actual users being in the millions smacks of arrogance -- they violated general stats laws on testing to significance. Heck, we do more A/B testing on throwaway sites than they did on THE search engine...sad.

Joe, the post above from BillyS addresses your point - this isn't the same thing as A/B testing where you get the testers for free - not for $75 an hour.

I used to conduct market research, this is a very big sample for this type of test. This is qualitative research, not quantitiative...

I'm not a fan of many things Google is doing, but it's important, IMO, to see things clearly and not with automatic anti-Google prejudice. That prejudgment undermines our ability to understand what's going on with the giant online influence.

I get that it is a different type of test Tedster (I won't totally agree that this is how a test should be rolled out, but will let that argument fall to the side for now); sample size is not enitrely disregarded in qualitative research though as it is important to have a large enough focus group in order to reduce the failure of discovery rate ~ my point is that group of 80 non-googlers should have been higher...to bring the rate to less than .001 probablity of missed discovery I think the size was supposed to be roughly 200 (it has been a little while). Quirk's, the journal on market research, has an article that delves into this more.

Side note: $75/hr with 200 people testing for a month straight without sleeping wouldn't even be a rounding error for their R&D. :)