Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| spider adds url? |
12 | smokeybarnable | 1:01 am Apr 24, 2008 | ||||
| Basalt |
4 | wilderness | 11:55 pm Apr 22, 2008 | ||||
| MLBot from Metadatalabs |
3 | incrediBILL | 11:41 pm Apr 22, 2008 | ||||
| TestBot yet another dodgy bot crawling from Amazon |
3 | Mokita | 8:06 pm Apr 22, 2008 | ||||
| My Latest Google, MSN, Yahoo Spider IP List IP list of GoogleBot, MSNBot and Yahoo Slurp spiders. CIDR formatted. |
3 | hdogan | 11:27 pm Apr 18, 2008 | ||||
| yangboz |
10 | wilderness | 6:26 pm Apr 17, 2008 | ||||
| darxi/1.1 New little creapy crawly |
3 | Ocean10000 | 2:01 pm Apr 17, 2008 | ||||
| Default User Agents Discussion |
7 | wilderness | 7:01 pm Apr 16, 2008 | ||||
| Yahoo! Slurp/3.0 released on new IPs Headaches for those Validating by IP instead of Reverse DNS |
12 | Receptional_Andy | 3:29 am Apr 16, 2008 | ||||
| HttpGetFile |
3 | Hobbs | 8:37 pm Apr 14, 2008 | ||||
| Exabot-Thumbnails Exalead |
5 | Bewenched | 7:48 pm Apr 13, 2008 | ||||
| Optioncarriere / Careerjet job search engines with generic ua's |
16 | incrediBILL | 6:43 pm Apr 13, 2008 | ||||
| Powerset who's adopting? |
10 | Hobbs | 8:38 am Apr 13, 2008 | ||||
| Scraper from Micro Synergy |
6 | keyplyr | 4:43 am Apr 13, 2008 | ||||
| bloglines crawler? |
5 | Bewenched | 4:25 am Apr 13, 2008 | ||||
| Are users visiting my site from Google.com? Some browser like agents are visiting my site from Google.com |
17 | schnee | 4:12 am Apr 13, 2008 | ||||
| Open ISearch/1.2.1 |
3 | Bewenched | 4:01 am Apr 13, 2008 | ||||
| cameleon scraper bot Different UA for every hit |
9 | Hobbs | 9:28 am Apr 12, 2008 | ||||
| Cidr |
2 | wilderness | 8:01 pm Apr 11, 2008 | ||||
| Who or what is ip-78-137-163-***.dedi.digiweb.ie |
5 | Drreggae | 8:38 am Apr 11, 2008 | ||||
| YodaoBot Beijing,China |
4 | Bewenched | 3:39 am Apr 10, 2008 | ||||
| baidu |
4 | wilderness | 6:05 am Apr 8, 2008 | ||||
| verifying ia archiver & Yahoo Bot via RDNS |
4 | Hobbs | 9:59 am Apr 7, 2008 | ||||
| Is this really a PlayStation 3 How can you tell if the browser is really a PlayStation 3? |
12 | Ocean10000 | 3:25 am Apr 7, 2008 | ||||
| Quick primer on identifying bot activity. And a how to guide to slow and stop scraping |
21 | Ocean10000 | 12:48 pm Apr 4, 2008 |
