Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Lexibot Allow or ban |
4 | mack | 11:38 pm Oct 10, 2002 | ||||
| rabaz Seems well behaved |
2 | carfac | 5:19 pm Oct 10, 2002 | ||||
| ua: ht |
3 | carfac | 4:50 pm Oct 10, 2002 | ||||
| What is contype UA? |
5 | transistor | 3:55 pm Oct 10, 2002 | ||||
| What is Alexa? |
3 | palmpal | 5:45 am Oct 10, 2002 | ||||
| YComp 5.x.x.x in so many user agents what the heck is it? |
10 | GaryK | 11:25 pm Oct 6, 2002 | ||||
| Ywo new- and different- ones spotted today |
6 | carfac | 7:46 pm Oct 6, 2002 | ||||
| Microsoft Internet Explorer/4.40.426 Obvious - yet very strange - fake browser UA |
5 | Dreamquick | 6:10 pm Oct 4, 2002 | ||||
| SlySearch? |
22 | hanuman | 3:21 pm Oct 4, 2002 | ||||
| $botname Someone loose a variable? |
3 | carfac | 2:18 pm Oct 4, 2002 | ||||
| MARTINI - sysadmin@looksmart.net Is this a new L$ robot or just someone who likes their liquor? |
10 | Dreamquick | 7:58 am Oct 4, 2002 | ||||
| HTTrack 3.0x what is this? |
5 | allybongo | 8:48 pm Oct 3, 2002 | ||||
| Whois "-" "-"? How do I block "-" "-" from my Robots.txt file? |
11 | jrota | 5:35 pm Oct 3, 2002 | ||||
| Need a Bit of a Tutorial... Trying to figure out what it is... |
3 | carfac | 8:04 pm Oct 2, 2002 | ||||
| Web Download Tool Something else to ban |
12 | carfac | 6:39 pm Oct 2, 2002 | ||||
| We weirdest UA Ever! |
5 | carfac | 7:23 pm Oct 1, 2002 | ||||
| New Fast IP |
3 | volatilegx | 5:12 pm Oct 1, 2002 | ||||
| Another new one! Lickity_Split |
6 | carfac | 3:23 am Sep 30, 2002 | ||||
| What is w.x.y.z.nombres.ttd.es What engine is this a spider from? |
3 | bnhall | 1:21 pm Sep 28, 2002 | ||||
| Scooter/3.2.SF0 |
2 | Scooter24 | 8:13 pm Sep 27, 2002 | ||||
| www.go2-yellowpages.com Deep crawling for awhile now, and incorrectly |
4 | savvy1 | 8:11 pm Sep 27, 2002 | ||||
| Spiders/Bots and Javascript Spiders without javascript indexing our "javascript required" page |
6 | Advantex | 8:46 pm Sep 25, 2002 | ||||
| Caught another "Trademark Search" robot |
3 | carfac | 6:44 pm Sep 25, 2002 | ||||
| antibot-V1.1.5/i586-linux-2.2 belongs to AntiSearch |
8 | volatilegx | 5:32 pm Sep 25, 2002 | ||||
| NetMechanic V3.0 Seems to be spidering? |
2 | Nick_W | 8:56 am Sep 24, 2002 |