I'm curious as to if anyone knows how the spyfu's, keycompete, keywordspy , googspy etc. type applications query google thousands and thousands of times per day to record keywords, ads, urls etc.

will they have a commercial licence or something from google to do this, ie whitelisted IP range or doing it via some API route? or if not they must be using some workaround to avoid being blocked after a thousand or so searches as I understand is the limit for commecrial applications..

By the looks of it these applications are crawling for data non stop so I'm assuming surely it must be licenced activity!?