Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Strange new "no-referrer" traffic |
16 | Scarecrow | 4:05 pm Nov 18, 2008 | ||||
| Mozilla/5.0 (compatible; OpenX Spider; http://www.openx.org) Posed as this then crawled as Nutch |
9 | GaryK | 5:43 pm Nov 17, 2008 | ||||
| WebTV Ballmer's rival for YouTube? |
6 | Samizdata | 3:51 am Nov 17, 2008 | ||||
| Bots Attack Using Randomized User Agents Simply Blocking Libwww-PERL Won't Work |
12 | incrediBILL | 1:06 am Nov 17, 2008 | ||||
| 1813 how you're doing with it today? |
4 | smallcompany | 10:38 pm Nov 16, 2008 | ||||
| BeB-cart Why is a shopping cart crawling my site? |
3 | GaryK | 5:33 pm Nov 15, 2008 | ||||
| lnbot FAST Enterprise Crawler 6 used by LexisNexis |
4 | caribguy | 2:19 pm Nov 12, 2008 | ||||
| Japanese AV bot? |
3 | Megaclinium | 9:02 pm Nov 10, 2008 | ||||
| copyright sheriff Another rights enforcer? |
3 | GaryK | 10:59 pm Nov 9, 2008 | ||||
| is this Googlebot legit? "Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)" |
6 | Nkona | 7:31 pm Nov 9, 2008 | ||||
| Mozilla/5.0 (compatible; Google Keyword Tool; +https://adwords.google. Anyone see this bot crawling lately |
14 | trinorthlighting | 3:15 pm Nov 9, 2008 | ||||
| Fake Googlebot from SoftLayer creepy creepy crawly crawly |
26 | Samizdata | 11:36 pm Nov 8, 2008 | ||||
| Google Home Page as Referrer and Random User Agents Is this a scraper? |
9 | dataguy | 9:00 pm Nov 8, 2008 | ||||
| WebDataCentreBot Mozilla/5.0 (compatible; WebDataCentreBot/1.0; +http://WebDataCentre.com/) |
9 | Receptional_Andy | 10:07 pm Nov 5, 2008 | ||||
| GoogleBot Crawl Frequency What's a likely lag time for Google to crawl new title tags? |
6 | SamNiccolls | 7:50 pm Nov 3, 2008 | ||||
| What is mvk-it.com crawler? |
2 | rivergirl | 5:23 pm Nov 2, 2008 | ||||
| Top reasons to ban bots |
6 | Clark | 11:16 pm Oct 29, 2008 | ||||
| SEOmoz Crawler/Bot [2] ( 1 2 ) |
49 | g1smd | 9:20 pm Oct 29, 2008 | ||||
| Slurp now fetching CSS Is this a new behavior? |
3 | jdMorgan | 5:13 pm Oct 28, 2008 | ||||
| MSN Live |
2 | wilderness | 3:44 pm Oct 28, 2008 | ||||
| GoogleT5 |
5 | keyplyr | 1:44 pm Oct 21, 2008 | ||||
| Bots find new domain in days robot/scrape activity before there's a site |
9 | dstiles | 5:19 pm Oct 20, 2008 | ||||
| Why is there no mention of BotBuster on WW? Good or bad, there should be some mention, right? |
15 | MichaelBluejay | 12:26 am Oct 17, 2008 | ||||
| Webmaster World Websensed Websense is blocking access to webmasterworld.com for many corporate users. |
2 | spotter | 11:32 pm Oct 15, 2008 | ||||
| Slurp is turning bad. |
7 | waynne | 11:19 pm Oct 15, 2008 |