Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Ask with no user agent lots of hits |
5 | Bewenched | 1:36 pm Aug 24, 2007 | ||||
| PageFetcher-Google-CoOp; |
6 | keyplyr | 8:57 pm Aug 22, 2007 | ||||
| ASK with no reverse DNS and no UA |
13 | Romeo | 3:29 am Aug 21, 2007 | ||||
| Spider running from Qwest |
7 | btherl | 1:27 am Aug 21, 2007 | ||||
| compute.amazonaws.com sucked down entire site - maybe twice |
2 | amznVibe | 9:31 pm Aug 17, 2007 | ||||
| ozzie |
3 | wilderness | 1:47 pm Aug 17, 2007 | ||||
| France Telecom? but really resolves to Performance Systems International Inc. |
11 | Bewenched | 7:23 am Aug 6, 2007 | ||||
| Gigabot not identifying itself? |
2 | Bewenched | 5:41 pm Aug 4, 2007 | ||||
| Is this googlebot or mediapartners? Same IP, different host resolution? |
3 | johnnie | 5:41 pm Aug 4, 2007 | ||||
| http://127.0.0.1:4664/preview?event id= |
5 | frontpage | 12:29 am Aug 3, 2007 | ||||
| HTMLParser/1.6 from research27.corp.sp1.yahoo.com |
4 | blend27 | 8:43 pm Aug 2, 2007 | ||||
| What am I doing wrong guys to block IP,s rDNSbot help re - IP,s |
2 | whatsdoin | 2:05 pm Aug 2, 2007 | ||||
| IBM Supercomputer : Blueice Scraping my site |
5 | RayWatson | 6:24 pm Jul 30, 2007 | ||||
| Ban All Nutch Variants? |
2 | keyplyr | 5:34 pm Jul 29, 2007 | ||||
| Banning Googlebot Spoofers? |
5 | keyplyr | 4:25 am Jul 19, 2007 | ||||
| Suspicious computer from Microsoft IP range pretends to come from Search Live results - what's this? |
3 | radix | 5:04 am Jul 11, 2007 | ||||
| Is this really googlebot feedfetcher? agent feedfetcher |
2 | Clark | 1:02 am Jul 7, 2007 | ||||
| Gigabot IP address? |
8 | keyplyr | 11:43 pm Jul 4, 2007 | ||||
| BoardReader-Image-Fetcher |
4 | wilderness | 5:15 am Jun 25, 2007 | ||||
| facebookexternalhit/1.0 |
2 | MaxM | 3:13 am Jun 25, 2007 | ||||
| shelob v1.0 - No robots.txt From Juniper Networks |
4 | jdMorgan | 7:00 pm Jun 17, 2007 | ||||
| IEAutoDiscovery User-Agent DeskShare AutoDiscovery Active Web Reader RSS feed reader |
2 | VisitingPaul | 4:53 pm Jun 14, 2007 | ||||
| Is this really ia archiver? What is the true IA user agent? |
4 | bouncybunny | 4:15 pm Jun 14, 2007 | ||||
| worio+bot+heritrix/1.10.0 worio+bot+heritrix/1.10.0 |
6 | blend27 | 2:25 am Jun 13, 2007 | ||||
| Intelix/0.7 New bot from Amazon? |
3 | GaryK | 8:27 pm Jun 12, 2007 |