Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| New MSN bot? adidxbot |
12 | dstiles | 6:54 pm Jul 6, 2009 | ||||
| ovale.ru's: "ovalebot3.ovale.ru facepage" and "search new sites" 1 host, 2 bots, 0 robots.txt |
2 | Pfui | 6:18 pm Jul 5, 2009 | ||||
| solfo-linkchecker If it looks like a bot... |
2 | Pfui | 6:57 pm Jul 3, 2009 | ||||
| Tagoobot/3.0 http://www.tagoo.ru |
4 | tpeacock | 1:48 am Jul 1, 2009 | ||||
| "slurp, yahoo! slurp, slurp/2.0, inktomi slurp, slurp.so/1.0" Is someone at Yahoo playing some sort of joke? |
2 | Pfui | 10:41 pm Jun 30, 2009 | ||||
| ShrinkTheWeb.com Crawler |
2 | Pfui | 8:44 pm Jun 30, 2009 | ||||
| Rexyobot |
10 | Pfui | 8:41 am Jun 30, 2009 | ||||
| WideCircles Analysis Agent |
2 | Pfui | 12:37 am Jun 30, 2009 | ||||
| Spider Hunting or Site Building? Goes in waves, do you trust your bot blocker? |
8 | incrediBILL | 9:26 pm Jun 28, 2009 | ||||
| smart.apnoti.com Robot/v1.34 |
3 | keyplyr | 5:53 pm Jun 28, 2009 | ||||
| DoubleVerify Crawler |
4 | keyplyr | 9:01 am Jun 26, 2009 | ||||
| Pinboard (pinboard.cantbedone.org) |
4 | Pfui | 3:34 am Jun 26, 2009 | ||||
| HolmesBot (http://holmes.ge) |
6 | Pfui | 4:20 pm Jun 23, 2009 | ||||
| Zanran ZanranCrawler/0.2 (helpdesk@zanran.com) |
3 | Frank_Rizzo | 8:22 am Jun 22, 2009 | ||||
| WordPress support page bot? |
5 | GaryK | 4:19 am Jun 22, 2009 | ||||
| msnbot using IE UA should i suspect screenshots? |
3 | GaryK | 11:21 pm Jun 21, 2009 | ||||
| acont.de's ACONTBOT |
15 | Pfui | 11:04 pm Jun 21, 2009 | ||||
| Baidu Behaving Badly: Goes undercover w/ cloaked UA; omits robots.txt "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)" |
8 | Pfui | 9:47 pm Jun 21, 2009 | ||||
| Mahalobot/1.0 |
3 | keyplyr | 5:51 pm Jun 20, 2009 | ||||
| googlebot head request anyone seen googlebot performing head requests? |
16 | enigma1 | 2:18 pm Jun 18, 2009 | ||||
| MSIndianWebcrawl |
8 | Pfui | 7:11 am Jun 16, 2009 | ||||
| SpiderMan/1.0 |
5 | Pfui | 4:07 am Jun 16, 2009 | ||||
| Something new from Yandex? VisualParser |
8 | GaryK | 4:57 am Jun 14, 2009 | ||||
| AppEngine-Google |
4 | dstiles | 10:24 pm Jun 13, 2009 | ||||
| Servage Robot |
2 | caribguy | 3:11 pm Jun 8, 2009 |