Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| new spiders to me anyone know why two new spiders but why theyare crawling |
2 | chompy | 7:44 am Apr 15, 2003 | ||||
| is 64.171.96.89 a bot? ip spam on server |
4 | indiandomain | 6:24 am Apr 15, 2003 | ||||
| HTTPLib/1.0 whats this guy? |
2 | Andrue | 5:28 am Apr 15, 2003 | ||||
| no UA but then a: Mozilla/5.0 (Macintosh UA Mozilla/5.0 (Macintosh |
4 | Andrue | 4:53 am Apr 15, 2003 | ||||
| PF Free Web Search Tool Another one to ban...<sigh> |
5 | pendanticist | 4:14 am Apr 15, 2003 | ||||
| Way to "block" userAgents with ASP.NET Just a little script I have written.. |
6 | kewlbeezer | 6:37 pm Apr 14, 2003 | ||||
| spiders agent names |
11 | bignooz | 10:37 am Apr 14, 2003 | ||||
| Do you guys ban Cyveillance? They don't behave very well sometimes |
9 | kewlbeezer | 10:21 am Apr 14, 2003 | ||||
| Thai spider, or something else? wierd hits from asianet.co.th |
7 | Ankheg | 2:47 am Apr 14, 2003 | ||||
| Mac Finder 1.0.39 How about this crawler? |
7 | kewlbeezer | 11:58 pm Apr 13, 2003 | ||||
| Googlebots being spoofed? How can you tell? |
14 | pendanticist | 9:41 pm Apr 10, 2003 | ||||
| List of 'BAD' bots? Which one are they? |
3 | le_gber | 8:57 pm Apr 10, 2003 | ||||
| Mozilla/3.0 (compatible) Again Pulling strange urls |
5 | jimbeetle | 8:17 pm Apr 10, 2003 | ||||
| 65.92.240.232 - Is this a Googlebot? Googlebot/1.0 |
8 | Jesse_Smith | 4:04 pm Apr 10, 2003 | ||||
| URL_Spider_Pro/3.0 Update to URL Spider Pro/1.92 |
2 | pendanticist | 5:42 am Apr 10, 2003 | ||||
| proxy.google.com |
2 | coffeelover | 12:49 am Apr 10, 2003 | ||||
| timboBot/0.9 weblog indexer |
2 | frontpage | 10:33 pm Apr 9, 2003 | ||||
| 'index.html' and 'index.htm' Will spiders see both if they are both in root directory? |
2 | TWhalen | 9:51 pm Apr 9, 2003 | ||||
| bad bot or bad self made spider trap? |
6 | matthias | 6:30 pm Apr 9, 2003 | ||||
| snorkel-x.markwatch.com Is this a spider? |
5 | scone | 6:04 pm Apr 9, 2003 | ||||
| AlkalineBOT behaving a bit aggressive... |
7 | RonPK | 9:17 am Apr 9, 2003 | ||||
| what's this (PHP/4.2.3) |
5 | fandangood | 7:58 am Apr 9, 2003 | ||||
| What to do with names and I.P. addresses of spiders? why spend a ton of time gathering info? |
11 | Brad1ey | 1:06 am Apr 9, 2003 | ||||
| Windows NT 5.2? I wasn't aware of this type of browser... |
15 | kewlbeezer | 5:11 pm Apr 8, 2003 | ||||
| Weak bot 80.8.120.54 |
3 | wilderness | 12:45 am Apr 8, 2003 |