Since this thread is too old
[
webmasterworld.com...]
Checks Robots.txt: Yes
Ignores Robots.txt: No, but it's a bit slow to read and start obeying it.
Behavior: It had been hitting my main page and a JavaScript file a few times a day, when it started going a bit mad on the 31st of August and requested the main page 99 times in a day.
On the 1st of September it behaved more normally (4 request) but on the 2nd 99 requests.
At this point I had enough and updated robots.txt to ban it.
3rd of September another 94 hits (and didn't read robots.txt all day).
4th of September, read robots.txt, and then proceeded to get the main page another 84 times.
5th of September, didn't visit, and ever since has just requested robots.txt and gone away.