Forum Moderators: open
Notes: In addition to robots.txt, the only accurate, real-file hits are marked [okay]. The site is Yahoo-authenticated and Site Explorer's allowed URL list is accurate.
-----
llf320021.crawl.yahoo.net
Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; [help.yahoo.com...]
02:27:44 [okay]
02:28:55 [okay]
02:47:13 /robots.txt
02:49:26 /dir/list.php?show=4
02:51:09 /dir/list.php?show=38
02:52:14 /dir/list.php?show=5885
02:52:44 /dir/list.php?search=LAFUMA
02:53:13 /dir/list.php?show=30
02:54:01 [okay]
02:54:51 /dir/contact.php
02:55:21 /dir/list.php?show=9
02:55:51 /dir/list.php?show=5
02:56:21 /dir/list.php?show=3902
02:56:51 /dir/list.php?expand=10
02:57:21 /dir/list.php?show=28
02:57:51 /dir/list.php?search=Flow
02:58:21 /dir/list.php?search=Atomic
02:58:51 /dir/list.php?show=36
02:59:21 /dir/list.php?show=853
02:59:51 /dir/list.php?expand=2
03:00:51 /dir/list.php?expand=3309
03:01:21 /dir/list.php?show=22
03:02:21 /dir/list.php?search=HIGH%20PEAK
03:02:51 /dir/list.php?search=ELAN
03:03:51 /dir/register.php
03:04:21 /dir/list.php?show=4851
03:04:51 /dir/list.php?search=Tyrolia
03:05:21 /dir/list.php?expand=20
03:05:51 /dir/list.php?show=51
03:06:21 /dir/forum
03:06:51 /dir/list.php?show=5866
03:07:21 /dir/list.php?show=27
03:15:50 [okay]
03:20:12 [okay]
03:31:24 /dir/list.php?show=3903
-----
P.S. Posted here rather than in "Yahoo Search Engine and Directory" because those convos seemed more about SERPs than Slurp activity. Apologies if in wrong place.
P.P.S. Congrats, IncrediModBILL! :)
Personally I am [ this ] close to banning Yahoo all together.
You should be ashamed Hobbs ;)
Where your sense of history, tradition and dedication ;)
Don
I am [ this ] close
Don, I'm all for history and tradition but I also expect the same from them. ;o)
FWIW: 10-plus hours in, Slurp 3 is still hitting away at nonexistent "list.php" files -- in between legit files -- even after I smacked it with mod_rewrite four hours ago. (sighs) Then again, Slurp China drops by umpteen times/day and it's been rewritten for years...
FWIW Redux: I've already blocked all Y crawlers but Slurp (regular) and Slurp DE. And I'm this close to blocking the latter because .de is right up there on my Countries Spawning Bad Bots list.
Is anyone else blocking Slurp 3.0, and/or Slurp DE? Any noticeable drop in SERPs or good traffic?
no one's sure of the eventual effect(s).
We're sure of the eventual effects: ZERO YAHOO TRAFFIC!
I would check my traffic stats before making such a rash decision.
(Since this is new (mis)behavior, I figured I'd block Slurp 3.0 for a while, then open it back up and watch. We shall see.)
I have valid expiry headers. I have a valid and accurate sitemap.xml. No other bots screw up like this. For a major player, they are extremely rude and egocentric when it comes to respecting our properties.
Apparently Yahoo! Slurp DE is the crawler for a (D)irectory (E)ngine service that crawls preferred content explicitly listed by Yahoo! Search content service partners.Slurp DE will respect robots.txt rules for User-Agent: Slurp DE or User-Agent: Yahoo! Slurp DE. If those user agents are not listed Slurp DE will obey User-Agent: Slurp.
"REAP-crawler Nutch/Nutch-1.0-dev (Reap Project; [reap.cs.cmu.edu...] Reap Project)"
nice choice of name 5 reap and 2 nutch in one UA?
On the page it says:
The REAP crawler is a web robot that sifts the web looking for documents that can be used by the REAP project, a research project at Carnegie Mellon University that develops software to help people that are learning English to improve their vocabulary skills
I just got it, this is not Yahoo, but a site hosted at Yahoo, this could get confusing, anyone has the Yahoo client hosting Ip ranges?
