I'm not clear why you are calling this a "user agent string" bug. If your server returned a 500 http status, then your server was saying it encountered an unexpected internal condition that prevented it from answering the request. Is your server configured to treat different user agents differently?

it appears that way. The only requests that were resulting in a 500 were ones from googlebot and it was the user agent string that appeared to have been different.

What was the offending user-agent string, then?

Jim

User-Agent: Mozilla/5.0
instead of User-Agent: Mozilla/4.0

Admin note: Here's a link to a good summary of the issue:

[kowitz.net...]

[edited by: tedster at 6:34 pm (utc) on Nov. 7, 2007]

Reading the fix I found from your search it appears that it was ASP.Net 2.0 choking on the "Mozilla/5.0" part of the string -- not really google-related problem at all. And it wasn't limited to just googlebot, but a few other UAs ASP.NET also choked on. It's tough, but sounds like if you did put the recommended fixes in the only thing you can do now is sit back and wait until your pages are reindexed.

I've been researching this for several clients on the .NET 2.0 platform, and those who use Helicon's ISAPI Rewrite for url rewriting do not seem to have a problem with googlebot's US string. I haven't zeroed in yet on which solutions do trigger the 500 error problem, or why. A number of .NET/IIS rewriting solutions are discussed on Scott Gu' blog [weblogs.asp.net] (he's a developer for Microsoft) and only some of them have the problem.

I'd sugggest that anyone using .NET 2.0 install Firefox and the UserAgentSwitcher add-on so they can try a page from their site as Googlebot and discover if they get a 500 error. It's a quick health check-up that is well worth the little bit of time it takes.

I also note that mikey158's problem "fixed itself" with no apparent changes on his server - so possibly the Google crawl team is taking some action here when they see too many 500 http errors. All I can imagine them doing, however, is re-crawling with a different user-agent string, one that does not include "Mozilla".

Does anyone see that in their server logs?

Anyone hazard a guess why Microsoft has not fixed this yet? Its been a while since Google changed their user agent...

Do you have IIS setup to use its internal Compression, or a 3 party compression by chance?

Could this be addressed by updating the browsecap.ini file to include this flavor of mozilla or am I misunderstand the use of browsecap?

The suggestion on the page linked above [kowitz.net] is:

create a "genericmozilla5.browser" file in your "/App_Browsers" folder in the root of your application... This will match generic Mozilla compatible browsers and spiders with user-agents strings such as Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

There are not bugs in ASP.NET

Bugs are in your code.......

I banged my head against the wall for weeks with this bug. (And yes, it's an ASP.NET 2.0 bug.)

There is a simple work-around:

1. In your ASP.NET's App_Browsers directory, add a file called BrowserFile.browser

2. Add the following to the BrowserFile.browser file:

<browsers>
 <browser refID="Mozilla">
  <capabilities>
   <capability name="cookies" value="true" />
  </capabilities>
 </browser>
</browsers>

Pete

How do i reproduce this bug.
I have the whole site written on ASP.NET 2.0 and do not see any problem with googlebot.

What do i need to do to see that bug?

Just set up Firefox with the UserAgentSwitcher add-on and change your user agent to googlebot's:

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Then go to your site. If you can browse it, then you don't have a problem with this bug.

The exact conditions that trigger this bug are still not 100% clear to me, and neither am I clear whether either Google or Microsoft have taken steps to try to compensate for it. The most I can say is that it seems to be triggered at times by using .NET's native url rewriting, for example, the HttpContext.RewritePath Method [msdn2.microsoft.com]

The third party re-write utilities such as ISAPI Rewrite do not seem to trigger it. But I do want to emphasize that I'm using language filled with weasel words like "seems to". I recommend people do this quick check with the Firefox add-on, or check your server error logs to make sure googlebot is not generating 500 error codes.

I had same issue for over 4 months and never figured out why my sites doesn't get indexed by google. last night i found out! Like you've said google always got 500 response and reason was server side advert request code of one of the advertises i use.

":" in google bot's user agent string was breaking it!

I can vouch for tedster's supposition: for me the 500 error occurred when I was performing URL mapping using Context.Rewritepath within my ASP.NET's Application_BeginRequest code in Global.aspx, such as this:

Context.RewritePath("~/Default2.aspx", False)

The workaround I posted above, adding a .browser file to your asp.net's App_Browsers directory, was recommended by Microsoft after I submitted the bug to them in early 2006. The problem went away completely and I have seen no side effects at all.

I would think that others in this community would have experienced this problem bigtime, considering that many of us use Context.Rewritepath for SEO purposes.

That said, I am unable to recreate the problem today with the test case that I submitted to Microsoft. So, my assumption here is that a patch may have come out for ASP.NET that has addressed the problem. (I have since upgraded to .NET 3.0, maybe that's what did it.)

Pete

I have a site which uses ASP only, and the 500 error is occuring. I have installed the Firefox extension, and when I visit the page as Googlebot, it works fine, however the log records a 500 error! Can anyone explain that?

My website is ASP.net compatible however, so could I still create the browser file in App_Browsers? (i have just tried this and the error occurs - though i cant restart the site manually to properly test this).

Chris.

Kill me. I had switch user agent to Googlebot and went to my site.
Everything works perfectly.
and I do use HttpContext.RewritePath to make my urls SE friendly.

So still think that you people have a bug somewere......

Thanks George - hope we can move to more clarity here. I note in pclaar's post above that Microsoft confirmed a bug affecting his case. I also note that you are not seeing the problem in you case. Seems there must be another factor around that is involved in tripping the problem.

Yes, Microsoft *confirmed* that this was a bug in ASP.NET 2.0 and offered me the workaround I listed previously. I had created a very simple test case that had two web pages and one line of code in the global.aspx that caused the error when the User_Agent string was set to Googlebot or Ask Jeeves/Teoma. The error that caused the 500 was "Cannot use a leading .. to exit above the top directory."

I believe that this bug has been fixed, because I cannot get it to reproduce anymore. This bug was over a year old, and I was running on ASP.NET 2.0 in early 2006. After a few 2.0 patches and the installation of .NET 3.0, I can no longer reproduce the error.

I'd be curious to know if Mickey158 has installed .NET 3.0 or at least has patched 2.0. I think that alone might fix his issue instead of the workarounds.

Pete