Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| WebDAV-MiniRedir |
2 | wilderness | 1:26 am Feb 24, 2009 | ||||
| GrubNG Is this a Vista crawler? |
2 | tangor | 5:06 pm Feb 23, 2009 | ||||
| A pair of WordPress crawlers No idea what purpose they serve |
5 | GaryK | 2:37 am Feb 23, 2009 | ||||
| spider appending url? |
2 | smokeybarnable | 11:34 pm Feb 21, 2009 | ||||
| Do you see a lot of "google.co.jp" referers, looking more like a bot? I'm seeing a lot of "random" traffic from 122.208.209.nnn |
3 | kzim | 1:38 am Feb 20, 2009 | ||||
| WWW-Mechanize/0.72 Sneaky |
6 | tangor | 8:06 am Feb 19, 2009 | ||||
| Kintiskton Banned after I stopped laughing |
21 | caribguy | 4:56 am Feb 19, 2009 | ||||
| happyfunsearch.com HappyFunBot |
2 | jmccormac | 5:24 pm Feb 16, 2009 | ||||
| Yanga WorldSearch Bot |
18 | keyplyr | 2:28 pm Feb 16, 2009 | ||||
| Opera.com Bot Opera/9.10 (Windows NT 5.1; U; en) |
4 | Pfui | 10:12 pm Feb 15, 2009 | ||||
| Spider or ripper? |
5 | tangor | 10:10 pm Feb 15, 2009 | ||||
| Safari favicon bot? This is really weird! |
5 | tangor | 8:50 pm Feb 15, 2009 | ||||
| LucidMedia ClickSense |
7 | wilderness | 8:10 pm Feb 13, 2009 | ||||
| Same IP switching UAs, over 7,000 404 errors |
4 | smallcompany | 11:57 pm Feb 12, 2009 | ||||
| Revisiting "VB Project": Kick or Keep? |
3 | Pfui | 11:25 pm Feb 12, 2009 | ||||
| Google Wireless Transcoder |
6 | dstiles | 3:15 am Feb 11, 2009 | ||||
| Does Google have a mobile crawler? If so do I have the right UA?[2] ( 1 2 ) |
38 | GaryK | 12:24 am Feb 8, 2009 | ||||
| nasty distributed crawler |
3 | gurgeous | 12:01 am Feb 8, 2009 | ||||
| Obfuscated bot? Phone? Any code-breakers in the house? |
3 | Pfui | 11:52 pm Feb 7, 2009 | ||||
| Indeed.com spider name? |
3 | stephenph | 10:59 pm Feb 5, 2009 | ||||
| Alexa.com Bot - ai archiver What is this? |
5 | ginar | 10:08 pm Feb 4, 2009 | ||||
| DealGates Bot/1.1 From France? ovh.net |
3 | caribguy | 8:58 pm Feb 4, 2009 | ||||
| Iac |
2 | wilderness | 12:16 am Jan 31, 2009 | ||||
| librabot/1.0 ( http://search.msn.com/msnbot.htm) Something new from MSFT |
7 | GaryK | 12:26 am Jan 29, 2009 | ||||
| Googlebot uses Godaddy IP range |
26 | keyplyr | 12:05 am Jan 29, 2009 |