Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| BeijingCrawler from -- Massachusetts (64.95.76.33); ignores robots.txt IP = Teragram Corp. Ran TeragramCrawler last month. |
4 | Pfui | 4:09 pm Jun 25, 2006 | ||||
| West Wind Internet Protocols 4.xx Never seen this before |
20 | zCat | 3:01 am Jun 25, 2006 | ||||
| Exalead Thumbnail bot |
6 | fiestagirl | 4:21 am Jun 24, 2006 | ||||
| Writing a bot for my site to check links. Is robots.txt validation required? |
18 | Duskrider | 6:52 am Jun 23, 2006 | ||||
| Gigabot UA change? dropped versioning |
5 | jdMorgan | 6:45 am Jun 23, 2006 | ||||
| Announcement: TrueLocal TUCKER 0.1 Mozilla/5.0 (compatible; TUCKER/0.1; +http://www.truelocal.com/tucker.aspx) |
4 | bakedjake | 9:24 pm Jun 22, 2006 | ||||
| Mozilla (Google Web Accelerator Cache Warmer; Google-TR-1) IP belongs to Road Runner |
9 | GaryK | 7:27 pm Jun 22, 2006 | ||||
| "mail.visvo.com" runs trio o' bots: Anonymous; NutchCVS; Skywalker TLD "visvo.com" nameservers = .yahoo.com (!) |
8 | Pfui | 5:56 pm Jun 22, 2006 | ||||
| EmeraldShield crawled a disallowed folder |
13 | Mokita | 2:18 am Jun 22, 2006 | ||||
| Spider tracking Tools to analyse crawling |
3 | rupalis | 12:26 am Jun 22, 2006 | ||||
| Bot coming from Everyones Internet |
9 | Mokita | 12:00 am Jun 22, 2006 | ||||
| User Agent: Ken |
2 | fusion5 | 8:34 pm Jun 20, 2006 | ||||
| FDSE robot |
5 | bull | 1:18 am Jun 20, 2006 | ||||
| Snappy/1.1 urltrends bot |
4 | Mokita | 6:04 am Jun 19, 2006 | ||||
| Nutch Sightings from 100+ IPs |
18 | incrediBILL | 3:51 am Jun 19, 2006 | ||||
| I'm a newbie Know what this is? |
5 | jcmoon | 5:03 pm Jun 15, 2006 | ||||
| what's wrong with my sitemap.xml sitemap |
4 | youriv | 10:56 pm Jun 14, 2006 | ||||
| WISEbot/1.0 (WISEbot@wisenut.co.kr; http://wisebot.wisenut.co.kr) No robots.txt |
12 | GaryK | 11:26 pm Jun 12, 2006 | ||||
| exactseek.com Last forum thread seems to be from 2002 |
5 | GaryK | 6:09 pm Jun 12, 2006 | ||||
| Teoma's new IPs a direct hit... |
14 | incrediBILL | 11:32 pm Jun 11, 2006 | ||||
| Worio don't worio, be happy |
5 | incrediBILL | 8:26 pm Jun 11, 2006 | ||||
| Yahoo? Overture? mozilla/4.0 |
10 | fiestagirl | 12:54 pm Jun 11, 2006 | ||||
| Looksmart crawling with Mozilla/4.0 User-agent Doesn't identify itself |
4 | jdMorgan | 7:30 pm Jun 9, 2006 | ||||
| Strange? Just a heads up |
4 | wilderness | 6:27 am Jun 8, 2006 | ||||
| mailer.hiphopcaucus.net Anyone see it? |
5 | youfoundjake | 6:21 am Jun 8, 2006 |
