Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| v92 .nat.svl.searchme.com spider.? taking bandwith? |
7 | jim_knopf | 7:21 pm Jun 19, 2008 | ||||
| AutoProxy4 |
3 | wilderness | 10:36 pm Jun 17, 2008 | ||||
| Mozilla/5.0 (compatible; ScoutJet; http://www.scout jet.com/) Letting it in, or blocking it? |
4 | g1smd | 6:54 pm Jun 16, 2008 | ||||
| Is Cogentco = Twiceler? they're back again... |
3 | peterg22 | 3:35 pm Jun 16, 2008 | ||||
| Baidu Block |
6 | outland88 | 9:10 am Jun 13, 2008 | ||||
| LinkScanner, AVG, Trend Micro, 1813 and SV1 Disambiguation of secretive anti-virus tools |
21 | Samizdata | 2:49 pm Jun 10, 2008 | ||||
| Suspicious crawling Any ideas what this is? |
2 | Thez | 10:23 am Jun 10, 2008 | ||||
| amazonaws goes stealth |
2 | incrediBILL | 7:05 pm Jun 9, 2008 | ||||
| AOL changing IPs Velocity and other traps |
2 | phred | 10:23 pm Jun 5, 2008 | ||||
| yoofind yoono webcrawler new look |
11 | Hobbs | 4:27 pm Jun 3, 2008 | ||||
| Trend Micro AV May Be Causing Excess Traffic [2] ( 1 2 ) |
31 | blend27 | 9:25 pm Jun 2, 2008 | ||||
| Kalooga |
13 | keyplyr | 4:41 pm Jun 1, 2008 | ||||
| Are all UAs ending in "SV1)" invalid? |
9 | Mokita | 3:01 pm Jun 1, 2008 | ||||
| Sitemaps.xml Single requests, various IPs |
2 | Umbra | 5:08 pm May 30, 2008 | ||||
| Google IP Requesting Images No user agent for the images |
7 | incrediBILL | 5:36 am May 30, 2008 | ||||
| Separate Google HOST and ADDR run Java/1.6.0 01 No robots.txt no matter what. |
2 | Pfui | 1:53 pm May 28, 2008 | ||||
| Googlebot from Serbia? |
6 | mrjones | 2:59 am May 28, 2008 | ||||
| Corporation Service Company? |
3 | smokeybarnable | 9:24 am May 22, 2008 | ||||
| OpenAcoon? |
4 | smokeybarnable | 7:16 pm May 21, 2008 | ||||
| alef/0.0 |
2 | keyplyr | 10:10 pm May 19, 2008 | ||||
| Searchme Who are they? |
5 | fischermx | 5:38 pm May 19, 2008 | ||||
| Infohelfer/0.9 |
3 | incrediBILL | 5:37 pm May 17, 2008 | ||||
| discobot discoveryengine |
20 | Hobbs | 12:38 pm May 6, 2008 | ||||
| OnetSzukaj bot out of poland - hungry one too |
2 | Bewenched | 3:28 pm May 4, 2008 | ||||
| Generic Bot Filtering Criteria What keywords do you use to filter? |
16 | wilderness | 6:10 pm May 3, 2008 |