Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| MetagerBot/0.8-dev (MetagerBot; http://metager.de; ) Note space before close paren |
8 | Pfui | 7:19 pm Jul 25, 2006 | ||||
| Downloads from a blank UA? |
23 | keyplyr | 6:26 am Jul 25, 2006 | ||||
| HA! No User Agent for You! These just flat-out annoy me! |
10 | GaryK | 2:16 am Jul 25, 2006 | ||||
| Crawler/1.0 http://elibron.com No robots.txt |
5 | GaryK | 7:24 pm Jul 24, 2006 | ||||
| Mozilla/5.0 (compatible;MAINSEEK BOT) No robots.txt |
3 | GaryK | 3:06 pm Jul 24, 2006 | ||||
| Mozilla/5.0 (compatible; robtexbot/1.0; http://www.robtex.com/ ) Note space before close paren. Also: no robots.txt; uses site URL in ref |
10 | Pfui | 1:50 am Jul 24, 2006 | ||||
| "teoma agent1" from directhit.com -- no robots.txt |
2 | Pfui | 11:17 pm Jul 22, 2006 | ||||
| "research-spider" from .cs.brown.edu |
2 | Pfui | 11:16 pm Jul 22, 2006 | ||||
| "Entrieva/1.0" -- no robots.txt |
2 | Pfui | 10:26 pm Jul 22, 2006 | ||||
| 000s of Truncated Page Requests from Many IPs [3] ( 1 2 3 ) |
82 | jomaxx | 10:59 pm Jul 20, 2006 | ||||
| Yahoo! Crawlers - A response from Yahoo! Search Response from Yahoo! |
9 | Yahoo_Mike | 8:33 am Jul 18, 2006 | ||||
| How to ban (compatible ; type requests Note space between compatible and semicolon[2] ( 1 2 ) |
40 | larryhatch | 3:06 am Jul 18, 2006 | ||||
| Googlebot Google but not Googlebot |
4 | vortech | 4:55 pm Jul 16, 2006 | ||||
| server2.attributor.com |
12 | Cromicon | 4:39 pm Jul 16, 2006 | ||||
| Naughty Yahoo User Agents Please post them here[2] ( 1 2 ) |
45 | GaryK | 5:04 pm Jul 14, 2006 | ||||
| New UA: Mozilla/4.0 (compatible; mark.blonin.bot;) has a weird Referer |
6 | Mokita | 6:44 pm Jul 13, 2006 | ||||
| sna-0.0.1 mikeelliott@hotmail.com (nee mikemuzio@msn.com) Spider as Snoopy Spammer |
2 | Pfui | 1:13 am Jul 13, 2006 | ||||
| NASA Search 1.0? |
6 | keyplyr | 6:00 pm Jul 12, 2006 | ||||
| Feed43 Proxy/1.0 (www.feed43.com) |
2 | zCat | 2:35 pm Jul 10, 2006 | ||||
| A Python-urllib hit 8000+ times yesterday |
2 | bigreat | 8:16 pm Jul 9, 2006 | ||||
| SBIder/0.8-dev Anyone getting this? |
15 | youfoundjake | 3:21 pm Jul 8, 2006 | ||||
| Hadrinka Tumaj Al-Kahal WebWasher 3.4 |
9 | Pfui | 6:18 am Jul 7, 2006 | ||||
| Yahoo Mindset No exclamation mark! |
3 | GaryK | 5:35 am Jul 5, 2006 | ||||
| page_verifier Scanning for malware |
5 | GaryK | 10:19 pm Jul 3, 2006 | ||||
| Redirect Test/0.1 from Amazon.com |
5 | GaryK | 10:10 pm Jul 3, 2006 |