Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Lipperhey Link Explorer www.lipperhey.com |
4 | grandma_genie | 7:39 am Nov 28, 2010 | ||||
| 80legs 80legs abuse |
4 | MxAngel | 3:23 pm Nov 27, 2010 | ||||
| UA blocking of 80legs bot? using .htaccess to block 80legs bot |
8 | classifieds | 7:13 am Nov 27, 2010 | ||||
| PostRank |
3 | JAB_Creations | 7:03 pm Nov 26, 2010 | ||||
| Multiple User Agents after one page at the same time. Twitterbot, Topsy, Tweetmemebot |
5 | grandma_genie | 9:22 am Nov 21, 2010 | ||||
| Is this a legitimate User Agent? Mozilla/4.0 |
8 | grandma_genie | 12:15 pm Nov 19, 2010 | ||||
| Ask Jeeves-Teoma Cutbacks to cut crawling? |
9 | Pfui | 7:32 pm Nov 18, 2010 | ||||
| WebGo IS - 5724 |
13 | keyplyr | 9:37 pm Nov 11, 2010 | ||||
| Now seeing Bingbot Testing the waters a few days early[2] ( 1 2 ) |
31 | jdMorgan | 5:59 pm Nov 11, 2010 | ||||
| crawler4j |
4 | incrediBILL | 5:46 am Nov 9, 2010 | ||||
| pirst pirst; MSIE 8.0; |
2 | Dijkgraaf | 9:31 am Nov 6, 2010 | ||||
| SockrollBot |
3 | Pfui | 10:00 pm Nov 5, 2010 | ||||
| FyberSpider FyberSpider/1.3 (http://www.fybersearch.com/fyberspider.php) |
2 | Dijkgraaf | 1:52 am Nov 4, 2010 | ||||
| Kindle Qs |
2 | Pfui | 10:20 pm Nov 1, 2010 | ||||
| dedibox.fr Badly behaved bot returns only slightly better behaved. |
4 | Dijkgraaf | 12:22 am Nov 1, 2010 | ||||
| Requests for logfiles Whois DE |
4 | caribguy | 12:19 am Nov 1, 2010 | ||||
| RankFlex.com Webspider |
2 | incrediBILL | 12:12 am Nov 1, 2010 | ||||
| Sosospider how to stop it? |
15 | smallcompany | 5:49 am Oct 29, 2010 | ||||
| semrush/whorush web marketing tools |
3 | Dijkgraaf | 4:39 am Oct 29, 2010 | ||||
| Which Proxy headers do you capture? There are so many |
3 | Clark | 11:11 pm Oct 27, 2010 | ||||
| Fake iPhone and iPad UA requests? Possibly a botnet |
3 | jdMorgan | 4:18 pm Oct 27, 2010 | ||||
| RiverglassScanner |
4 | keyplyr | 10:50 pm Oct 26, 2010 | ||||
| pzncrawl |
2 | Pfui | 10:01 pm Oct 26, 2010 | ||||
| Lots of Traffic from Rochester, NY IP addresses located in Rochester, NY either through RoadRunner or Frontier |
4 | suga | 5:26 pm Oct 26, 2010 | ||||
| Webairinternet2 |
4 | keyplyr | 10:53 pm Oct 25, 2010 |