I thought that people might be interested in something I am recently experiencing with Google. I just want to put this out there in case it helps someone, it is another data point that people can use when trying to figure out their own problems - this is not a general issue with Google, this is something I screwed up myself.

I have a system that prevents 'bots from crawling my site. It has a whitelist, to which I add Google IPs. I had always added them manually because new IPs didn't come up too often, and I wanted to make sure that no one was spoofing Google. About 10 days ago, Google apparently switched to crawling from about a dozen new IPs. I was not paying close attention to my system and those IPs got blocked. They were blocked for about 3 or 4 days.

On March 31, I noticed a slight downturn in traffic. I also noticed that my "traffic sources" was a little off - my Analytics has pretty consistently pegged my sources as 82-84% Organic, 11-13% Direct, and 4-5% Referral within a percentage point or two. But on March 31, I was down to 81% Organic. Just a tiny drop, but I hadn't been below 82% for months. I also graph some real-time metrics that show the amount of site usage, and that was down a bit too.

On a hunch, I started digging into things and noticed that I had blocked a bunch of Google crawlers. I quickly unblocked them. I didn't think much of it.

By about 1pm, I saw some Twitter chatter. People were wondering why, when they searched for a hockey player, my site wasn't returned when it had been in the past. I checked, and sure enough, they were right. Not for all players, but for a lot of them, especially the more popular players.

I have made boneheaded mistakes in the past, accidentally noindexing some pages, and when I did a "fetch as Google" and "Submit to index", Google recrawled and the page was added back within minutes. So that's what I did with some of my more popular pages. Unfortunately, Webmaster Tools listed that about 55,000 pages had been blocked. Considering that you can only "clear" 1,000 pages per day, I knew I had some work to do over the next two months - dutifully going into WMT and clearing that list, and that I would never be able to add all those pages back via the "fetch as Google" feature since there is a 500-page per month limit. I knew I would have to just take my medicine.

By the evening, a Reddit thread was started, asking why my site wasn't coming up for searches. And the twitter chatter continued. See, the thing is, even though I have a pretty prominent search box all over my site, people simply prefer to put a hockey player's name into Google along with my site's name, and they then click on the link in Google. And that was annoying users who either thought my site was gone, or who just don't want to change their routines.

The next day, my Analytics stats showed more of a drop - I was down to 79% Organic, 15% direct, and 7% referral. That persisted for 2 days, and my overall traffic was off by about 30%.

The traffic picked up a little bit, but slowly. Google wasn't adding the pages back even though they had recrawled them. Some pages came back, but some of my top pages (for example, Connor McDavid) were nowhere to be found in Google - even when I searched with my site's name (as many users do). I tried asking Google to recrawl multiple times, but after a week they still aren't adding back pages for which I request a recrawl.

I also noticed that although I had 5 straight days of recovery, on the 6th day, I got knocked down a peg again. It could have been seasonality - my traffic goes up and down based on hockey game schedules - but it seems more like a Google induced issue to me. But on the 7th day, I'm still going up, gradually.

Here are some odd things I noticed. First, although Google tells me that over 80% of my traffic is "Organic", and that figure dropped by only about 4 percentage points, it doesn't add up to me mathematically that my overall traffic would be down by 20-30%.

Next, my advertising revenue is way off. This is harder to pin down because the issue took place over a month boundary, and ad rates and fill percentages often change over monthly boundaries, but my effective CPM dropped by almost 2/3 even though my traffic was initially down by less than 1/3 and is currently down by about 20%. My effective CPM is not rising even as my traffic rises. This could be due to a couple of things:

Advertising is just lousy in April.

Advertising systems use the first couple of days of the month to "prime" their systems in some way, so if your traffic on those days is not representative, it skews the rest of your month.

Google is somehow sending a different "blend" of visitors, and this group is not behaving the same way with respect to ads. In other words, they are sending visitors less likely to click on an ad.

The visitors that Google are still not sending are very valuable to the advertisers, perhaps because most only view one or two pages (when people go to my site directly, they are usually there for a while - they are the more loyal and in-depth visitors).

I don't think there is anything I can do here except wait this out and patiently explain to people on Twitter that yes, I screwed up; yes, I fixed my end, and that we just have to wait for Google to finish its course of action. However I just found the whole thing interesting, academically, because I don't think anyone would ever deliberately do this as a case study.