Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Who is Kosmix? New crawler sighted in server logs. |
14 | grandma_genie | 10:02 pm Aug 21, 2010 | ||||
| zerozero88 |
3 | MxAngel | 12:34 am Aug 21, 2010 | ||||
| Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:x.x.x) Gecko/200411 Fake UA |
2 | Dijkgraaf | 4:27 am Aug 20, 2010 | ||||
| Hailoobot New Search Engine for Middle East |
3 | Dijkgraaf | 2:06 am Aug 19, 2010 | ||||
| Ukrainian Academic and Research Network Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC |
2 | Dijkgraaf | 10:28 pm Aug 17, 2010 | ||||
| extrabot |
2 | Pfui | 2:12 am Aug 17, 2010 | ||||
| New Bot On The Block IBM Bot Named SAI-Crawler |
3 | SevenCubed | 6:10 am Aug 16, 2010 | ||||
| bixolabs bot broken 1433 loads of the same URL over 21 hours |
6 | rowan194 | 5:04 am Aug 15, 2010 | ||||
| How to block all robots except google, yahoo, msn, and ask |
7 | vphoner | 7:58 pm Aug 14, 2010 | ||||
| Mozilla 5.0/Metal will never die |
2 | Dijkgraaf | 7:51 pm Aug 14, 2010 | ||||
| SemtoBot/0.1 |
2 | keyplyr | 3:32 pm Aug 12, 2010 | ||||
| Opera/9.80 (Windows NT 5.1; U; ru) Presto/2.6.30 Version/10.60 Russian scraper |
2 | Dijkgraaf | 2:03 am Aug 12, 2010 | ||||
| bitmagicbot |
10 | Pfui | 5:06 pm Aug 10, 2010 | ||||
| Why the obsession with Favicon? |
8 | Mokita | 12:31 pm Aug 10, 2010 | ||||
| Python-httplib2 |
2 | yaashul | 9:55 pm Aug 9, 2010 | ||||
| carrier-enabler |
3 | Pfui | 4:42 pm Aug 8, 2010 | ||||
| Yandex bots - incorrect rDNS? |
7 | dstiles | 9:40 pm Aug 6, 2010 | ||||
| How to Identify bad bots from legitimate users Need help on hints and tips that give away a bad bot |
11 | liam_85 | 3:43 pm Aug 6, 2010 | ||||
| Google%20Quick%20Search%20Box |
4 | Pfui | 12:29 pm Aug 4, 2010 | ||||
| CustomUserAgent Apple surprise? |
12 | Pfui | 6:16 pm Aug 1, 2010 | ||||
| Teleport Ultra |
3 | Pfui | 11:14 pm Jul 31, 2010 | ||||
| ARIN Whois |
11 | wilderness | 1:24 pm Jul 31, 2010 | ||||
| How to trap bots? That crawls disallowed pages. |
25 | Sunnz | 2:16 pm Jul 26, 2010 | ||||
| Botnet Attempting Zen Cart Attack Blank UAs Attacking |
13 | incrediBILL | 3:08 am Jul 24, 2010 | ||||
| wtv bot |
2 | Pfui | 3:40 am Jul 23, 2010 |
