Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| How to ban a crawler that I can't see I know I'm being crawled, but I can't find the bot |
15 | arieng | 1:54 am Jun 30, 2010 | ||||
| MSNbot changing to Bingbot on Oct.1, 2010 |
3 | incrediBILL | 10:24 pm Jun 29, 2010 | ||||
| Twenga-Bot |
2 | tangor | 5:36 am Jun 29, 2010 | ||||
| Box-Agent |
3 | Pfui | 10:02 pm Jun 28, 2010 | ||||
| binlar (larbin) From an old pest: 207-67-117-17* |
2 | Pfui | 10:04 pm Jun 26, 2010 | ||||
| CogentCo |
9 | keyplyr | 10:42 pm Jun 17, 2010 | ||||
| How to enable every spider from Google, Yahoo and Bing ? How to bann every other spider/bot/robot ! |
12 | Future | 12:26 pm Jun 16, 2010 | ||||
| Awfully quiet in Spiders lately. |
4 | incrediBILL | 12:20 am Jun 16, 2010 | ||||
| GoogleToolbar in User Agent Does this imply it's not readng the page? |
19 | dstiles | 9:11 pm Jun 12, 2010 | ||||
| eSobi - opinions? desktop-based clipping app sends invalid HTTP_IF_MODIFIED_SINCE headers |
11 | caribguy | 1:48 am Jun 10, 2010 | ||||
| msramlbot (from Microsoft; no robots.txt; no thanks) |
2 | Pfui | 5:40 pm Jun 7, 2010 | ||||
| Facebook spidering? |
2 | Bewenched | 7:55 am Jun 5, 2010 | ||||
| SetEnvIf User-Agent |
7 | wilderness | 5:33 am Jun 5, 2010 | ||||
| WebSurfer text/* Who is this? |
4 | montclairguy | 5:33 am Jun 5, 2010 | ||||
| WordPress/MU Bot |
3 | dstiles | 9:42 pm May 30, 2010 | ||||
| Facbook spidering my site? been going on for about 4 hours now |
3 | Bewenched | 6:00 pm May 29, 2010 | ||||
| Gazopa Feast Far too many hits |
5 | dstiles | 9:44 pm May 28, 2010 | ||||
| Bing/1.1 CFNetwork/459 |
3 | wilderness | 6:38 pm May 24, 2010 | ||||
| Deluged with unknown bots |
19 | montclairguy | 5:56 am May 23, 2010 | ||||
| Unknown bot scrape or probe? |
6 | enigma1 | 8:05 am May 18, 2010 | ||||
| Googlebot now hiding its UA |
4 | enigma1 | 3:18 pm May 17, 2010 | ||||
| New archive.org UA includes heritrix :) |
13 | caribguy | 2:52 am May 17, 2010 | ||||
| .b2b3a |
7 | wilderness | 8:59 pm May 14, 2010 | ||||
| Missing HTTP HOST No value at all in HTTP_HOST |
9 | dstiles | 8:30 pm May 14, 2010 | ||||
| 80legs [3] ( 1 2 3 ) |
61 | GaryK | 7:18 am May 12, 2010 |