Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Google IP Requesting Images No user agent for the images |
7 | incrediBILL | 5:36 am May 30, 2008 | ||||
| Separate Google HOST and ADDR run Java/1.6.0 01 No robots.txt no matter what. |
2 | Pfui | 1:53 pm May 28, 2008 | ||||
| Googlebot from Serbia? |
6 | mrjones | 2:59 am May 28, 2008 | ||||
| Corporation Service Company? |
3 | smokeybarnable | 9:24 am May 22, 2008 | ||||
| OpenAcoon? |
4 | smokeybarnable | 7:16 pm May 21, 2008 | ||||
| alef/0.0 |
2 | keyplyr | 10:10 pm May 19, 2008 | ||||
| Searchme Who are they? |
5 | fischermx | 5:38 pm May 19, 2008 | ||||
| Infohelfer/0.9 |
3 | incrediBILL | 5:37 pm May 17, 2008 | ||||
| discobot discoveryengine |
20 | Hobbs | 12:38 pm May 6, 2008 | ||||
| OnetSzukaj bot out of poland - hungry one too |
2 | Bewenched | 3:28 pm May 4, 2008 | ||||
| Generic Bot Filtering Criteria What keywords do you use to filter? |
16 | wilderness | 6:10 pm May 3, 2008 | ||||
| HeartRails Capture Yet another Screen Capture Website |
4 | Ocean10000 | 3:59 pm May 3, 2008 | ||||
| PhpDig/1.8.8 New Zealand |
2 | Bewenched | 8:12 pm Apr 30, 2008 | ||||
| Yahoo! Slurp/3.0 used for iffy purposes? 26 hits to "list.php" files -- on non-PHP site |
13 | Pfui | 11:49 am Apr 29, 2008 | ||||
| oegp v. 1.3.0 a confirmed scraper |
3 | Hobbs | 5:47 am Apr 29, 2008 | ||||
| Stealth GoogleBot, or Spoofed IP? Strange Hits in Stats, WHOIS Shows they come from Google |
4 | abhorrent12 | 3:31 am Apr 28, 2008 | ||||
| unknown |
23 | wilderness | 11:26 pm Apr 26, 2008 | ||||
| Ask Jeeves/Teoma oddities odd IPs and reverse DNS |
8 | incrediBILL | 1:45 pm Apr 26, 2008 | ||||
| Strange hits from internetserviceteam.com |
5 | rudyten | 6:39 pm Apr 25, 2008 | ||||
| SharedService.Crawler resolves to Microsoft? |
2 | Bewenched | 4:44 pm Apr 25, 2008 | ||||
| Blocking the bad boys keeping bots, breakers and bugs out |
4 | superclown2 | 11:55 am Apr 24, 2008 | ||||
| spider adds url? |
12 | smokeybarnable | 1:01 am Apr 24, 2008 | ||||
| Basalt |
4 | wilderness | 11:55 pm Apr 22, 2008 | ||||
| MLBot from Metadatalabs |
3 | incrediBILL | 11:41 pm Apr 22, 2008 | ||||
| TestBot yet another dodgy bot crawling from Amazon |
3 | Mokita | 8:06 pm Apr 22, 2008 |
