function is_this_a_valid_web_crawler($remote_host_ip) { //This function should return TRUE as soon as possible since it's testing to see if an IP address belongs to a vaild web crawler.
if (is_this_a_real_msnbot($remote_host_ip)) {return TRUE;}
elseif (is_this_a_real_GoogleBot($remote_host_ip)) {return TRUE;}
elseif (is_this_a_real_Alexa_ia_archiver($remote_host_ip)) {return TRUE;}
elseif (is_this_a_real_ArchiveORG_ia_archiver($remote_host_ip)) {return TRUE;}
else {return FALSE;}
}

Don't you miss the Yahoo call
function is_this_a_real_YahooSlurp($remote_host_ip)
so it should read:

function is_this_a_valid_web_crawler($remote_host_ip) { //This function should return TRUE as soon as possible since it's testing to see if an IP address belongs to a vaild web crawler.
if (is_this_a_real_msnbot($remote_host_ip)) {return TRUE;}
elseif(is_this_a_real_YahooSlurp($remote_host_ip)) {return TRUE};
elseif (is_this_a_real_GoogleBot($remote_host_ip)) {return TRUE;}
elseif (is_this_a_real_Alexa_ia_archiver($remote_host_ip)) {return TRUE;}
elseif (is_this_a_real_ArchiveORG_ia_archiver($remote_host_ip)) {return TRUE;}
else {return FALSE;}
}

Also read THIS [webmasterworld.com] (not impacting directly on your functions.

Yikes! Yes henry0 you're spot on! Yes, the last few lines of the PHP script should read how henry0 posted them. Nice catch. <blush>

Side note: in our implementation of this script, we permit the below IP addresses unlimited crawl time and page requests/min on our website.

Any IP address not validating using above script or matching the below IP ranges is banned if it makes more than 3.6 requests per second or 15 page views per minute. For our site we've found usage outside these ranges is "unhuman" and/or appears abusive.

If you have lots of images on a page (resulting in more hits/page) or small pages that people are going to click though quickly, then you'd want to raise these numbers.

Warning: As mentioned above, banning visitors to your website by IP address carries some risk - if you ban say Google, Yahoo, etc (or even say a human reviewer from Yahoo/Google working from outside address normal IP address space) then you might get dropped from their index resulting in no visitors. Yes, no visitors = no problems. But no visitors = no company = no job. No problems !== always a good thing.

#GOOGLE
66.249.64.0-66.249.95.255

#YAHOO
68.180.128.0-68.180.255.255

#YANDEX (Russian search engine)
77.88.24.0-77.88.27.255

#CUILL TWICELER -- Cuill search engine w/Twiceler is an experimental robot.
38.99.13.121-38.99.13.126
38.99.44.101-38.99.44.106
64.1.215.162-64.1.215.166
208.36.144.6-208.36.144.10

#INKTOMI
74.6.0.0-74.6.255.255

We recently revised the script based on our realization that MSN's crawler can map to two host names (like "livebot-65-55-208-52.search.live.com" AND "msnbot-65-55-208-52.msn.com") and that the Teoma crawler is using IP addresses outside it's own address space (MCI).

function is_valid_search_engine_bot($ip_address_to_test) {
$valid_bot_hostnames=array();$valid_ip_addresses=array();

/*
* As more search engines allow webmasters to validate their bots, just add them to this list.
*/
$valid_bot_hostnames["YahooSlurp"]=array("prefix"=>"", "suffix"=>".crawl.yahoo.net");
// http://www.seroundtable.com/archives/013781.html
$valid_bot_hostnames["GoogleBot"]=array("prefix"=>"", "suffix"=>".googlebot.com");
// http://googlewebmastercentral.blogspot.com/2006/09/how-to-verify-googlebot.html
$valid_bot_hostnames["Alexa_ia_archiver"]=array("prefix"=>"", "suffix"=>".alexa.com");//Undocumented
$valid_bot_hostnames["Archive_ia_archiver"]=array("prefix"=>"", "suffix"=>".archive.org");//Undocumented
$valid_bot_hostnames["msnbot"]=array("prefix"=>"livebot-", "suffix"=>".search.live.com");
// http://blogs.msdn.com/livesearch/archive/2006/11/29/search-robots-in-disguise.aspx
$valid_bot_hostnames["Ask_Jeeves_Teoma"]=array("prefix"=>"", "suffix"=>".ask.com");//Undocumented but we have to use this since Ask/Teoma seems to crawl from outside it's own address space, crawling from two separate MCI address blocks.
//Just add more above this line

/*
* Until a search engine allows us to validate, the only thing we can do is whitelist their whole address space.
* You can add more IP addresses to this list
* When you enter a range of IP addresses, all IP addresses in the range will validate as valid bots!
*/
$white_listed_ip_addresses[]=array("starting_ip"=>"38.99.13.121","ending_ip"=>"38.99.13.126");//Cuill_Twiceler 
$white_listed_ip_addresses[]=array("starting_ip"=>"38.99.44.101","ending_ip"=>"38.99.44.106");//Cuill_Twiceler 
$white_listed_ip_addresses[]=array("starting_ip"=>"64.1.215.162","ending_ip"=>"64.1.215.166");//Cuill_Twiceler 
$white_listed_ip_addresses[]=array("starting_ip"=>"208.36.144.6","ending_ip"=>"208.36.144.10");//Cuill_Twiceler 
$white_listed_ip_addresses[]=array("starting_ip"=>"77.88.24.0","ending_ip"=>"77.88.27.255");//YANDEX (Russian search engine) IP address space
$white_listed_ip_addresses[]=array("starting_ip"=>"208.185.160.0","ending_ip"=>"208.185.160.255");//Ask Jeeves Teoma IP address space 
$white_listed_ip_addresses[]=array("starting_ip"=>"74.6.0.0","ending_ip"=>"74.6.255.255");//INKTOMI IP address space
//Just add more above this line

//Don't touch anything below this line//
function gethostbyaddrl($ip) {
 $rrs=dns_get_record(implode('.',array_reverse(explode('.', $ip))).'.in-addr.arpa.',DNS_PTR);
 $revnames=array();
 foreach($rrs as $rr) $revnames[]=$rr['target'];
 return (count($revnames)) ? $revnames : FALSE;
}

if (long2ip(ip2long($ip_address_to_test))===FALSE) {return FALSE;} //The IP address we were going to test isn't a valid IP address so how could it be a valid search engine bot?

foreach ($white_listed_ip_addresses as $ip_address_range) {
if (ip2long($ip_address_to_test)>= ip2long($ip_address_range["starting_ip"]) && ip2long($ip_address_to_test)<=ip2long($ip_address_range["ending_ip"])) {return TRUE;}
}

if (gethostbyaddrl($ip_address_to_test)===FALSE) {return FALSE;} //The IP address we were going to test doesn't resolve to any hostname so how could it be a valid search engine bot?

foreach ($valid_bot_hostnames as $bot_name => $valid_prefix_suffix) {
 foreach (gethostbyaddrl($ip_address_to_test) as $returned_hostname) {
if ($valid_prefix_suffix["suffix"]!=".search.live.com") { //If we're validating something other than msnbot then follow normal validation rules.
if ($valid_prefix_suffix["suffix"]==substr($returned_hostname, -strlen($valid_prefix_suffix["suffix"]))) {//Validate that the reverse DNS of the IP address we're testing matches requirements.
foreach (gethostbynamel($returned_hostname) as $real_ip) { ///Lastly validate that the hostname now points back to at least one IP address that matches the original IP address we're testing. See http://en.wikipedia.org/wiki/Forward_Confirmed_reverse_DNS
if ($real_ip==$ip_address_to_test) {
return TRUE;
}
}
} 
} else { //If we're testing for a valid msnbot then we have to use these special rules below.
$the_hostname_should_be=$valid_prefix_suffix["prefix"].str_replace(".", "-", $ip_address_to_test).$valid_prefix_suffix["suffix"]; //Turns "192.168.1.100" into "livebot-192-168-1-100.search.live.com"
if ($the_hostname_should_be==$returned_hostname) {//Validate that the reverse DNS of the IP address we're testing matches requirements.
foreach (gethostbynamel($returned_hostname) as $real_ip) { ///Lastly validate that the hostname now points back to at least one IP address that matches the original IP address we're testing. See http://en.wikipedia.org/wiki/Forward_Confirmed_reverse_DNS
if ($real_ip==$ip_address_to_test) {
return TRUE;
}
}
}
}
} //END foreach (gethostbyaddrl($ip_address_to_test)
}//END foreach ($valid_bot_hostnames
return FALSE; //IF there's some unplanned exception, then tell the user FALSE since no exceptions should ever happen.
} //END function is_valid_search_engine_bot. Do anyting you want after this line.

//This is how you use it.
if (is_valid_search_engine_bot("192.168.1.100")===TRUE) {
echo "192.168.1.100 is a valid search engine robot. Let it do whatever it wants. Don't ban it.";
} else {
echo "192.168.1.100 is a bad robot. Limit what it can do or ban it.";
}

[edited by: eelixduppy at 10:30 pm (utc) on Mar. 20, 2008]
[edit reason] fixed formatting [/edit]

Be aware that Google has a LOT more IP ranges active than the one block that you already listed.

I don't see any Google IP addresses listed? Am I missing something?

Fortunately, Googlebot validates by host name, not IP address, so Google can use any IP address range they please. The script above just confirms that the host name of the IP address ends in ".googlebot.com."

[googlewebmastercentral.blogspot.com...]

Hi,

As more search engines allow webmasters to validate their bots, just add them to this list

Question: How do you find out about new "white" bots?

thanks for sharing!

Good question. Here's what I do:

I use the above function to add bad IP addresses to a web server blacklist much like the HTTP Blacklist implementation described at: [projecthoneypot.org...]

So periodically I review the number of requests that have been blocked and group them by IP address. If I see hundreds of requests from a single IP address it means the bot either: (1) belongs to a hacker that will not give up or (2) it's a valid search engine that keeps on trying.

I can tell the difference between hackers and real Bots by simply looking up who owns the IP address.

We just caught Gigablast/Gigabot like this. Unfortunately, Gigablast doesn't seem to offer the semi-guaranteed reverse IP like Google does so you have to do something else. This above script allows you to white list Gigablast's IP address space which seems like an ok practice to me. I mean Gigabot shouldn't be crawling from scores of IP addresses (yet) like Google might.

Warning: In using the above script for several months now, we've seen several times where Google's IP address (and MSN's) didn't resolve to *.googlebot.com as instructed. This could have been caused by either:

1. Google brought on a new IP address to start crawling from and didn't wait long enough for the PTR domain record to populate across the Internet. In this case, it would mean that doing a NSLOOKUP on the IP of the new Googlebot would not return the expected *.googlebot.com at that instant. Doing the same NSLOOKUP a few hours later would return the correct *.googlebot.com address.

2. The script's connection to its DNS server timed out. "Huh?", you say? Let me explain: The way the above script is written (on purpose) is that if it cannot get a timely response from the name server, then it assumes that the IP in question is bad. The script was written this way because you really should always be able to get a response from a name server, especially if the IP address you're looking up is a good one. It is very possible that a hacker doesn't register a PTR record with his/her IP address. This causes the PTR lookup process (at least on Linux based systems) to hang and not respond very well. I'm not sure if the issue of distinguishing between "name server is off line" and "there's no registered PTR record belonging to this IP address" can be resolved in PHP5/Linux.

Until the script is better able to determine these error responses, please use this script carefully.

Here's the dates and times (California time) we saw the above happen:

Googlebot - 66.249.65.77 on Saturday, April 26, 2008 11:52 AM
Googlebot - 66.249.67.168 on Friday, April 11, 2008 2:47 AM
Googlebot - 66.249.70.219 on Thu 2/7/2008 11:23 PM
Googlebot - Google 66.249.84.12 on Sun 1/27/2008 8:25 AM
msnbot - 65.55.208.111 on Sat 4/26/2008 12:06 PM
msnbot - 65.55.208.58 on Fri 4/11/2008 2:51 AM

Thanks gmillikan;
it obviously is a never ending task...
which brings the following question
Are we not better of using the reverse
I mean filtering against a list of bad bots
rather allowing good bots for as you mentionned ip could and will change.

Henry0, I think that's a good thought however the universe of "bad robots" is bigger and more rapidly changing than the list of good robots . Friends of mine are usually willing to identify themselves and walk in sunny places but enemies cloak themselves and lurk in the shadows.

So it seems more effective to spend our time identifying and working together with the readily identifiable good robots instead of trying to chase the bad ones?

The quick fix to the above script would be to white list Google's, MSN's, etc address space while continuing to use the PTR validation feature most of the bigger SE's have started offering.

It's certainly worthy of a good discussion. I'd expect to see this up at Slashdot sometime.

Anyone else?