Forum Moderators: open
The automated YSM landing page checker is hitting our little server pretty hard. The pages are all pretty complex and rely on a few external XML feeds, hence a sustained fetching of 8 pages a second is making the system keel over.
How fast have you seen this go? Is our experience a glitch or is it normal for it to go so fast?
I'm guessing that Yahoo maybe are expecting us to have better web-serving resources due to having an innacurate advertiser status of Diamond (The highest status - maybe they think we're bigger than we are).
Has anyone fixed such a problem?
Thanks
When we first upload a few thousand terms, Y's bots practically bring our server to its knees with around 30 hits per second. When we make modifications to, say, 20-30 terms ... it gets even worse! The bots not only check the landing pages for the modified terms, they also go screaming through ALL of the rest of our terms ... even ones we deleted months ago!
Count yourself fortunate that you're only getting dinged 8 times per second.
<edit>Oh yeah ... the funny part is that you can't start banning their bot servers (and there are many of them in different IP blocks) to keep them from killing your server, because then you get the 'Site Broken' excuse for not making the mods or running the ads. Cool, huh?</edit>
[edited by: StupidScript at 5:43 pm (utc) on Oct. 4, 2006]
