Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| lanshanbot/1.0 ( http://search.msn.com/msnbot.htm) This one looks familiar |
3 | GaryK | 9:52 pm Jul 3, 2006 | ||||
| New Google IPs GSA Crawler |
4 | fiestagirl | 3:29 pm Jul 3, 2006 | ||||
| Same-second hits from nec-labs using Java and -- Ken If at first you don't succeed, do a switcheroo. |
6 | Pfui | 3:20 am Jul 2, 2006 | ||||
| gsa-crawler Whois = Google? |
6 | coconutz | 11:21 pm Jul 1, 2006 | ||||
| /a1b2c3d4e5f6g7h8i9/nonexistentfile.php |
11 | Umbra | 3:32 am Jul 1, 2006 | ||||
| Snapbot Anyone know what it is?[2] ( 1 2 ) |
33 | Mokita | 3:18 am Jul 1, 2006 | ||||
| Psycheclone Anyone heard of this one? |
20 | malachite | 8:51 pm Jun 30, 2006 | ||||
| Excluding all UA's with spider, bot, and crawl in them Is this too broad a brush to paint with? |
5 | cfx211 | 8:16 pm Jun 29, 2006 | ||||
| Mozilla/4.0 (compatible; Google Desktop) New user-agent? |
3 | referer | 6:32 pm Jun 29, 2006 | ||||
| OutfoxBot disobeys robots.txt |
2 | keyplyr | 5:25 pm Jun 27, 2006 | ||||
| KBeeBot |
2 | keyplyr | 5:17 pm Jun 27, 2006 | ||||
| Charlotte the spider anyone seen this one yet? |
22 | innocbystr | 8:08 pm Jun 26, 2006 | ||||
| WebarooBot was RufusBot |
2 | keyplyr | 6:04 pm Jun 25, 2006 | ||||
| bot/1.0 claims to be Microsoft |
5 | 4string | 4:42 pm Jun 25, 2006 | ||||
| BeijingCrawler from -- Massachusetts (64.95.76.33); ignores robots.txt IP = Teragram Corp. Ran TeragramCrawler last month. |
4 | Pfui | 4:09 pm Jun 25, 2006 | ||||
| West Wind Internet Protocols 4.xx Never seen this before |
20 | zCat | 3:01 am Jun 25, 2006 | ||||
| Exalead Thumbnail bot |
6 | fiestagirl | 4:21 am Jun 24, 2006 | ||||
| Writing a bot for my site to check links. Is robots.txt validation required? |
18 | Duskrider | 6:52 am Jun 23, 2006 | ||||
| Gigabot UA change? dropped versioning |
5 | jdMorgan | 6:45 am Jun 23, 2006 | ||||
| Announcement: TrueLocal TUCKER 0.1 Mozilla/5.0 (compatible; TUCKER/0.1; +http://www.truelocal.com/tucker.aspx) |
4 | bakedjake | 9:24 pm Jun 22, 2006 | ||||
| Mozilla (Google Web Accelerator Cache Warmer; Google-TR-1) IP belongs to Road Runner |
9 | GaryK | 7:27 pm Jun 22, 2006 | ||||
| "mail.visvo.com" runs trio o' bots: Anonymous; NutchCVS; Skywalker TLD "visvo.com" nameservers = .yahoo.com (!) |
8 | Pfui | 5:56 pm Jun 22, 2006 | ||||
| EmeraldShield crawled a disallowed folder |
13 | Mokita | 2:18 am Jun 22, 2006 | ||||
| Spider tracking Tools to analyse crawling |
3 | rupalis | 12:26 am Jun 22, 2006 | ||||
| Bot coming from Everyones Internet |
9 | Mokita | 12:00 am Jun 22, 2006 |