Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| HBZ-Digibib |
2 | Pfui | 10:45 pm Sep 3, 2010 | ||||
| Dom2Dom |
15 | Pfui | 11:10 pm Sep 2, 2010 | ||||
| thenewpush.com Server farm pushes bad/fake UAs |
2 | Pfui | 9:35 pm Sep 2, 2010 | ||||
| What is heritrix? Who is Darwin? Found in User Agent. Should I block these? |
12 | grandma_genie | 7:17 pm Aug 31, 2010 | ||||
| Who is Tweatbot? Found this bot all by itself in server logs. |
6 | grandma_genie | 3:43 pm Aug 31, 2010 | ||||
| Microsoft Bing Mobile SocialStreams Bot |
3 | Pfui | 12:28 am Aug 30, 2010 | ||||
| PSS-Bot |
2 | keyplyr | 10:19 am Aug 26, 2010 | ||||
| msnbot to bingbot? |
2 | keyplyr | 6:53 am Aug 25, 2010 | ||||
| crawl.mp3realm.org Another maggot. |
2 | jmccormac | 5:51 pm Aug 24, 2010 | ||||
| schluetersche German bot |
3 | Dijkgraaf | 4:21 am Aug 24, 2010 | ||||
| Who is Kosmix? New crawler sighted in server logs. |
14 | grandma_genie | 10:02 pm Aug 21, 2010 | ||||
| zerozero88 |
3 | MxAngel | 12:34 am Aug 21, 2010 | ||||
| Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:x.x.x) Gecko/200411 Fake UA |
2 | Dijkgraaf | 4:27 am Aug 20, 2010 | ||||
| Hailoobot New Search Engine for Middle East |
3 | Dijkgraaf | 2:06 am Aug 19, 2010 | ||||
| Ukrainian Academic and Research Network Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC |
2 | Dijkgraaf | 10:28 pm Aug 17, 2010 | ||||
| extrabot |
2 | Pfui | 2:12 am Aug 17, 2010 | ||||
| New Bot On The Block IBM Bot Named SAI-Crawler |
3 | SevenCubed | 6:10 am Aug 16, 2010 | ||||
| bixolabs bot broken 1433 loads of the same URL over 21 hours |
6 | rowan194 | 5:04 am Aug 15, 2010 | ||||
| How to block all robots except google, yahoo, msn, and ask |
7 | vphoner | 7:58 pm Aug 14, 2010 | ||||
| Mozilla 5.0/Metal will never die |
2 | Dijkgraaf | 7:51 pm Aug 14, 2010 | ||||
| SemtoBot/0.1 |
2 | keyplyr | 3:32 pm Aug 12, 2010 | ||||
| Opera/9.80 (Windows NT 5.1; U; ru) Presto/2.6.30 Version/10.60 Russian scraper |
2 | Dijkgraaf | 2:03 am Aug 12, 2010 | ||||
| bitmagicbot |
10 | Pfui | 5:06 pm Aug 10, 2010 | ||||
| Why the obsession with Favicon? |
8 | Mokita | 12:31 pm Aug 10, 2010 | ||||
| Python-httplib2 |
2 | yaashul | 9:55 pm Aug 9, 2010 |