Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| fr crawler |
2 | zerillos | 2:37 am Dec 4, 2007 | ||||
| RegExp Needed for Ugly UA strings I've seen it here somewhere |
11 | blend27 | 10:46 pm Dec 2, 2007 | ||||
| MSIECrawler |
3 | zerillos | 1:59 pm Dec 2, 2007 | ||||
| Laugh of the day |
8 | wilderness | 10:23 pm Nov 29, 2007 | ||||
| balihoo/nutch |
2 | jamiro | 10:07 pm Nov 29, 2007 | ||||
| Banning the Windows 98 user agent and stopping 60% of scrapers |
20 | Hobbs | 9:56 am Nov 26, 2007 | ||||
| What exactly is SnapPreviewBot? |
11 | schnee | 4:07 pm Nov 21, 2007 | ||||
| VeriSign |
4 | wilderness | 2:50 pm Nov 21, 2007 | ||||
| temasekholdings crawler |
4 | zerillos | 5:42 pm Nov 20, 2007 | ||||
| Multiple UA's |
7 | wilderness | 1:28 am Nov 20, 2007 | ||||
| Exabot exabot.com ban or not to ban |
2 | Bewenched | 6:53 pm Nov 15, 2007 | ||||
| Strange UA |
4 | wilderness | 3:39 pm Nov 13, 2007 | ||||
| Odd User Agent. Java? |
3 | Gibble | 2:56 pm Nov 6, 2007 | ||||
| Lexxe/Robot |
3 | engine | 12:58 am Nov 5, 2007 | ||||
| Searchme.com comes across as Abovenet Communications |
4 | Bewenched | 7:45 am Oct 31, 2007 | ||||
| New bot or fake? I haven't seen this one before |
4 | icedowl | 7:42 am Oct 31, 2007 | ||||
| is this a google bot/spider? |
3 | sandyk20 | 4:31 pm Oct 30, 2007 | ||||
| Mozilla/5.0 (compatible; Google Keyword Tool; +http://adwords.google.c No PTR |
3 | GaryK | 3:30 pm Oct 26, 2007 | ||||
| Google IP Range - blank UA and no Referrer Blank User Agent from Google IP range getting 403's |
2 | The_Contractor | 11:34 pm Oct 25, 2007 | ||||
| Where'd this come from? new google local search engine? |
2 | Import_Export | 9:46 pm Oct 25, 2007 | ||||
| What's Gigablast up to? trying to spider with 'languages' |
4 | malachite | 1:26 pm Oct 22, 2007 | ||||
| gigamega Fell into spider trap, no robot info at site |
5 | Receptional_Andy | 12:36 am Oct 18, 2007 | ||||
| archive.org bot/1.13.1x successor to ia_archiver? |
2 | zCat | 9:27 pm Oct 13, 2007 | ||||
| canadasearchbot |
5 | wilderness | 8:48 pm Sep 26, 2007 | ||||
| customer-reverse-entry huh? |
5 | Bewenched | 4:25 am Sep 26, 2007 |
