As a matter of simple fact, Google did make manual interventions, as the following quote from the WSJ article clearly shows:

On the most important issue, that of Google’s prized search engine, the FTC report said Google altered it to benefit its own services at the expense of rivals. The report said Google “adopted a strategy of demoting, or refusing to display, links to certain vertical websites in highly commercial categories.” --- [wsj.com...]

When you pick out "certain vertical websites" to demote, or refuse to display, that's manual intervention by any reasonable definition.

Unfortunately, some people will continue to deny the truth even when it stares them in the face.

> Googlebot crawling the sites, which is not illegal even if robots.txt is ignored.

Depends upon the Terms of Service.

Crawling isn't the issue - using Amazon and Yelps data to influence search results and then telling them they would be kicked out if it wasn't allowed appears to be the core issue. hiybbprqag anyone?

> robots.txt

Is not germane to any legal discussion. It was never adopted by any standards body.

even so: Amazon clearly has told Google not to use it's data by banning it from product reviews:

"User-agent: Googlebot"
Disallow: /product-reviews/B0069IY63Y"

[amazon.com...]

Amazon has now, did they earlier?

Although robots.txt has not been adopted by a standards body, it is accepted industry practice, which does have importance in some jurisdictions - although I do not know about the US, I suspect,as it is a common law country, its likely to be the same.

If robots.txt has no bearing on the case, then Google can do anything that is fair use, and, as I said earlier yelp do not even claim copyright in the reviews.

The applicability of site TOS is doubtful at best. Is there any legislation or case law suggesting that TOS are binding on a spider that cannot read or understand them?

The commissioners are appointed figure heads and are never referred to as staff, key or otherwise.

The commissioners may be appointed (just as cabinet members and Supreme Court justices are appointed), but they aren't "figureheads," and the staff work for them.

They're also attorneys. You can read their bis at:

[ftc.gov...]

> which does have importance in some jurisdictions

show many any legal case were talk of robots.txt was allowed into evidence any manner?

There are only two that were present during this period and it takes a majority to issue a probe into illegal activity. I did some quick searches and can't locate a list of the Commissioners during this period. There is probably some change that changed hands for this not to have been investigated further.

It seems like many large orgs are getting intense scrutiny into their illegal practices. It's Google's turn now.

Amazon has now, did they earlier?

[web.archive.org...]

Unfortunately, some people will continue to deny the truth even when it stares them in the face.

As I said, the FTC found that Google used some anti-competitive practices, and demanded changes.

And as the Wall Street Journal said:

FTC Chairwoman Edith Ramirez told a Senate committee that a majority of commissioners didn’t support a case against Google on any of the allegations under investigation.

The allegations did not come from consumers but were apparently made by:

Yelp, TripAdvisor Inc. and Amazon.com Inc

I see it as a squabble between rapacious corporations, none of which are strangers to anti-trust investigations.

No skin off my banana.

...

bwnbwn wrote:
There are only two that were present during this period and it takes a majority to issue a probe into illegal activity. I did some quick searches and can't locate a list of the Commissioners during this period. There is probably some change that changed hands for this not to have been investigated further.

That's the way I remember it too. Some political and corporate groups managed to hobble the FTC for an extended period of time by preventing the Commissioners vacancies from being filled.

I did some quick searches and can't locate a list of the Commissioners during this period

Source: FTC website:

Edith Ramirez (2010-present)
Julie Brill (2010-present)
Jon Leibowitz (2004-2013)
Maureen K Ohlhausen (2012-present)
J Thomas Rosch (2006-2013)

It was the first result when I searched on... ummm... Google.

Conspiracy theorists might prefer to use a different search engine just in case.

...

Edith Ramirez (2010-present)
Julie Brill (2010-present)
Jon Leibowitz (2004-2013)
Maureen K Ohlhausen (2012-present)
J Thomas Rosch (2006-2013)

Well I don't have time right now to do any searching, but unless I'm reading that wrong, two of those left in 2013.

Oops, Samizdata -- I misunderstood your post. Somehow I thought it was meant to be a current list. Sorry

In any case, I doubt if any members here are top experts in anti-trust laws, or know exactly what evidence was found in the investigation, or know what subsequently went on behind the scenes at the FTC. So I don't see how we can reach any definite conclusions on those subjects.

In my opinion. the real issue is Google's malfeasance in using dishonest manipulations to promote its own products in its search results.

It might have been better to have had two separate threads, one about the FTC, and another about Google's malfeasance and promises to avoid such practices in the future..

I doubt if any members here are top experts in anti-trust laws

Judging by their biographies, the FTC Commissioners who reported to the Senate that they "didn’t support a case against Google on any of the allegations under investigation" seem to know a thing or two about the subject.

I don't see how we can reach any definite conclusions

You mean like this one?

there was a whitewash, attempted coverup

...

So, having read as much of the report as I could find online, I can say with complete confidence that the majority of posters in this thread either did not read it, or skimmed too fast to understand it.

I don't mean to piss anyone off, just a statement of fact. There are a lot of assumptions about what the report is saying, or means to say, based on excerpts from the WSJ article that are contradicted by the actual report. After reading the report I believe that the WSJ article was intentionally inflammatory. They imply some things that are not supported by the data.

I'm not here to get into the argument, just to invite everyone to actually read the report. There are some interesting insights into search.

For instance, the process of algorithm tweaks based on feedback from multiple iterations of human focus group review... @Brett_Tabke, I think you're misunderstanding what's happening there. There is no evidence of manual manipulation of results. What they, apparently, did was try a variety of different --completely automated-- algorithms in pursuit of the goal of diversifying shopping related search results. They kept trying different algos until they found one that produced SERPs the focus groups responded positively to.

From a coding perspective this is the only thing that makes sense. You can't rely on manual scoring, it's not scalable. It might create really amazing results in a limited niche for a few weeks, or a few months, but the internet changes too fast for it to last, and over time you'd end up with an ungodly mess as humans slacked in ongoing maintenance. Not to mention you'd need a different focus group to keep up with each niche. Not even a company Google's size could afford that many employees.

On the other hand if you use a combination of data analysis and human interaction to tweak your algos until they work, and then just let them run (until the next overhaul)... well honestly there's no other option at the scale we're talking about here.

If Google were to start actively penalizing specific sites that they considered a competetive threat then I'd hope the FTC would step in because at that point no one else would be able to stop them. But what I just read is a report by the FTC concluding that is, specifically, not what's happening.

IanKelley wrote:
If Google were to start actively penalizing specific sites that they considered a competetive threat then I'd hope the FTC would step in because at that point no one else would be able to stop them. But what I just read is a report by the FTC concluding that is, specifically, not what's happening.

According to the WSJ article, the report says:

Google "adopted a strategy of demoting or refusing to display, links to certain vertical websites in highly commercial categories."

Other quotes from the WSJ article include:

the report said the company illegally took content from rival websites such as Yelp, TripAdvisor Inc. and Amazon.com Inc. to improve its own websites.

The 160-page critique, which was supposed to remain private but was inadvertently disclosed in an open-records request, concluded that Google’s “conduct has resulted—and will result—in real harm to consumers and to innovation in the online search and advertising markets.”

Google agreed to some voluntary changes to its practices.

Then-Chairman Jon Leibowitz said in a written statement at the time that Google’s voluntary changes deliver “more relief for American consumers faster than any other option.”

Based on all of these revelations, most of them coming directly from the report, it's really hard for me to understand how anyone can conclude that Google didn't do anything wrong. Or for that matter, why Google would have "agreed to some voluntary changes to its practices" if there were no reason for them to do so..

it's really hard for me to understand how anyone can conclude that Google didn't do anything wrong

Nobody in this thread has voiced any such conclusion.

...

Well Samizdata -- My only purpose in posting in this thread is to try to help get to the truth, or as near to the truth as we can, based on the available information. Sometimes I have to wonder what other members' purposes are.

My only purpose in posting in this thread is to try to help get to the truth, or as near to the truth as we can, based on the available information. Sometimes I have to wonder what other members' purposes are.

Right. When all else fails, try the ad hominem approach.

My only purpose in posting in this thread is to try to help get to the truth

So provide evidence for your very serious allegation against serving FTC Commissioners:

there was a whitewash, attempted coverup

And explain how this qualifies as a cover-up:

Then-Chairman Jon Leibowitz said in a written statement at the time that Google’s voluntary changes deliver “more relief for American consumers faster than any other option.”

It seems to be quite open and uncovered to me.

Sometimes I have to wonder what other members' purposes are.

Do let us know your conclusions.

...

Samizdata, I'm not going to respond to any more of your quibbles. I've got better things to do with my time. If you want to bring up all the arguable statements that people have made in this thread, including some that you made, you going to have to do it without me. I already told you what my purpose in participating in this thread is. You can accept it or not, I don't care.

While we kiddies know this is a STORY, as far as the WORLD is concerned (or politicians) this is a non-starter. Just like "net-neutering" (once called "net neutrality")

Fanbois can't be swayed, tinfoilers can't be swayed.

The story, however, is relevant and might be useful for future plans and projections for our own work on the web. YMMV

Well Tangor, this is the first time that Google's teflon has been dented. This is a story in WSJ - not some random blog. This is the real deal. Businesses, politicians, and ordinary folks read that news. It will have an effect. Both in policy making, as well as Googles actions.

anyone can read the report:
[graphics.wsj.com...]

aside: one cool thing in the report was the usage of the term SERP. (which was coined here on WebmasterWorld by moi ;-)

"VI. Conclusion:
Staff concludes that Google's conduct has resulted -- and will result -- in real harm to consumers and to innovation in the online search anad advertising markets. Google has strengtheded its monopolies over search and search advertising through anticompetitive means, and has forestalled competitors' and would-be competitors' ability to challenge those monopolies, and this will have lasting negative effects on consumer welfare."

This is the real deal.

Actually, it turned out to be an not-so-real deal. :-)

The actions taken by the commissioner was the best "not-so-real deal" $24+ million could buy.

The real deal is the coverage the commissioners' inaction is receiving now.

Well, since you're into conspiracy theories, you should remember that the Wall Street Journal is owned by Google's arch-enemy, Rupert Murdoch. For all we know, the WSJ made the whole thing up. :-)

Ah, more spin, love it, what conspiracy theory am I into?

Is it a theory that google spent $24+ million on lobbying?

Is it a theory that the commissioners acted contrary to the recommendation of staff?

Help me out here EG since you're going to label me as a conspiracy theorist, tell me which of my statements was theory and not fact.

WSJ isn't done yet:

[wsj.com...]

Google's access to high-ranking Obama administration officials during a critical phase of the antitrust probe is one sign of the Internet giant's reach in Washington. Since Mr. Obama took office, employees of the Mountain View, Calif., company have visited the White House for meetings with senior officials about 230 times, or an average of roughly once a week, according to the visitor logs reviewed by the Journal.

Google's knack for getting in the room with important government officials is gaining new relevance as scrutiny grows over how the company avoided being hit by the FTC with a potentially damaging antitrust lawsuit. Last week, the Journal reported that the FTC's competition staff concluded that Google used anticompetitive tactics and abused its monopoly power in ways that harmed Internet users and rivals.

On Nov. 6, 2012, the night of Mr. Obama's re-election, Mr. Schmidt was personally overseeing a voter-turnout software system for Mr. Obama. A few weeks later, Ms. Shelton and a senior antitrust lawyer at Google went to the White House to meet with one of Mr. Obama's technology advisers.

During Mr. Obama's 2012 re-election campaign, Google employees were the second-largest source of campaign donations to his campaign by any single U.S. company, trailing only Microsoft.

A former Google vice president, Megan Smith, is now the U.S.'s chief technology officer and a high-tech adviser to Mr. Obama.

The comments on that last piece were something a politically driven marketer could really sink their teeth into.

Anyone know if Bing is still working with "political brawler" Mark Penn? nytimes.com/2012/12/15/technology/microsoft-battles-google-by-hiring-political-brawler-mark-penn.html

It gets even scarier for Google if Microsoft hires Frank Luntz, if he is still available anymore
theatlantic.com/politics/archive/2014/01/the-agony-of-frank-luntz/282766/

Yep, it's like Samizdata said:

I see it as a squabble between rapacious corporations.

(Google, Microsoft, News Corp., you name 'em.)