Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| GlueText Spider/Crawler Located |
7 | incrediBILL | 5:29 pm Jan 29, 2011 | ||||
| Did The Wayback Machine Die in 2010? |
4 | incrediBILL | 5:14 pm Jan 24, 2011 | ||||
| Whitelisting just the good bots |
12 | Dexie | 3:11 am Jan 23, 2011 | ||||
| Can anyone add anything else to this ? |
12 | Dexie | 9:59 pm Jan 22, 2011 | ||||
| How to block Unwanted Traffic Generators? Traffic jumped 10 fold in 2 months -- I need to tackle this :| |
9 | MaxGrenk | 7:32 am Jan 19, 2011 | ||||
| Mozilla/4.0 (compatible; ICS) What does ICS stand for? |
5 | SteveWh | 11:02 pm Jan 17, 2011 | ||||
| ARIN-WHOIS again |
4 | wilderness | 10:41 pm Jan 9, 2011 | ||||
| Suggesting an improvement to Majestic's validation keys |
4 | Clark | 10:19 pm Jan 5, 2011 | ||||
| Ning coming with Feedburner UA |
2 | zerillos | 2:26 am Jan 5, 2011 | ||||
| Is this Googlebot's ip? |
7 | tabish | 12:17 am Jan 5, 2011 | ||||
| New bot from Google? |
9 | Dijkgraaf | 12:40 pm Dec 28, 2010 | ||||
| bah humbug bot also known as grinch bot |
2 | incrediBILL | 3:20 am Dec 27, 2010 | ||||
| Facebook Referer Trapped Is uiserver a common referer situation? |
7 | dstiles | 5:09 am Dec 22, 2010 | ||||
| Is this Yahoo crawling via Google? |
6 | GaryK | 5:24 am Dec 15, 2010 | ||||
| New Yahoo China bot range |
6 | dstiles | 1:53 pm Dec 14, 2010 | ||||
| Feedburner coming from non-Google IP |
5 | zerillos | 12:17 pm Dec 14, 2010 | ||||
| Bad Bot Blocking how to recognize bat bots and block them |
6 | jasimon9 | 4:42 pm Dec 13, 2010 | ||||
| I spotted Gigablast crawling - Haven't seen it for a while |
4 | engine | 12:00 pm Dec 12, 2010 | ||||
| Google Web Preview [3] ( 1 2 3 ) |
66 | Mokita | 11:55 pm Dec 11, 2010 | ||||
| Flight Deck Bot (experimental) Experiment fail |
12 | Dijkgraaf | 1:05 am Dec 11, 2010 | ||||
| YahooCacheSystem |
2 | JAB_Creations | 6:13 pm Dec 10, 2010 | ||||
| HuaweiSymantecSpider Privacy, ethics |
7 | Frank_Rizzo | 9:04 pm Dec 8, 2010 | ||||
| Discobot crawler How to stop this horrible bot? |
8 | grandma_genie | 2:08 am Dec 8, 2010 | ||||
| Subdomain indexed with robots.txt disallowing?! |
4 | zelv | 6:52 pm Dec 5, 2010 | ||||
| How to block referrers from entire TLD? Referrer spam is really a headache! |
3 | TechSense | 6:47 pm Nov 30, 2010 |