Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Googlebot spoofing |
6 | GaryK | 6:17 pm Oct 12, 2006 | ||||
| Yahoo to be added to list of bad bots? Ignoring Crawl Delay - up to 19 pages/sec |
5 | inbound | 1:08 am Oct 10, 2006 | ||||
| Cloaked robot from Ask.com No robots.txt, requests and malformed URLs |
5 | jdMorgan | 6:25 am Oct 9, 2006 | ||||
| Yahoo YSM bot being bad - Please help! |
3 | inbound | 8:16 am Oct 7, 2006 | ||||
| MindMapCrawler What's this? |
7 | keyplyr | 5:24 pm Oct 6, 2006 | ||||
| User-Agent: Nelian Pty Ltd - Spider v2.1 Owner claims he occasionally crawls as Google |
8 | GaryK | 12:39 pm Oct 6, 2006 | ||||
| RSS READER (mctwist@mail.dr-k.info).? |
2 | koka | 7:40 am Oct 6, 2006 | ||||
| InfoPath.1 Revisited |
2 | smells_so_good | 5:03 am Oct 6, 2006 | ||||
| Heads up for a voracious scraper |
5 | Mokita | 2:28 pm Oct 1, 2006 | ||||
| iearthworm yet another rogue crawler from Yahoo! |
5 | GaryK | 1:30 am Oct 1, 2006 | ||||
| TMCrawler reads robots.txt, and that's it... |
2 | bobothecat | 4:42 am Sep 30, 2006 | ||||
| Yahoo! spidering from new Class Cs |
4 | volatilegx | 2:42 am Sep 30, 2006 | ||||
| VerticalMatch this is a new one |
2 | volatilegx | 3:39 pm Sep 28, 2006 | ||||
| miniRank/2.0 (miniRank; http://minirank.com/; website ranking engine) Has anyone else seen it yet? |
5 | GaryK | 3:27 pm Sep 28, 2006 | ||||
| Hits from unidentified blog trackers where are these spiders from? |
5 | abates | 4:04 pm Sep 27, 2006 | ||||
| ShopWiki Very active little bot .. no referrals |
5 | Bewenched | 6:45 pm Sep 23, 2006 | ||||
| Definitive way to validate GoogleBot authenticity official word from Google |
8 | incrediBILL | 4:53 pm Sep 23, 2006 | ||||
| Bot banning consultants and/or bad bot banning services Do they exist? Help for those of us who haven't a clue |
16 | Webwork | 3:00 am Sep 23, 2006 | ||||
| panscient data services block them NOW or pay later - 200 megs a day! |
7 | amznVibe | 5:11 am Sep 22, 2006 | ||||
| seo bot from AOL? who is this new bot showing up |
3 | bhartzer | 10:57 pm Sep 21, 2006 | ||||
| Bot or no Bot no user agent odd requests |
2 | Bewenched | 4:25 pm Sep 20, 2006 | ||||
| Google Mobile Indexing Agent? no request for robots.txt |
3 | keyplyr | 10:49 pm Sep 18, 2006 | ||||
| Adwords - adbot HELP - can't find it! |
2 | howiejs | 3:14 am Sep 15, 2006 | ||||
| IEAutoDiscovery Pre-fetcher, feed reader, e-mail harvester, or what? |
2 | GaryK | 8:03 pm Sep 13, 2006 | ||||
| Who is TridentSpider/3.1? spider from Slovenia? |
3 | Bewenched | 4:25 pm Sep 12, 2006 |