These are the Linode ranges I block. Ever since they started their Cloud Servers there's been increasing trouble from every one of these ranges.

69.164.192.0 - 69.164.223.255
69.164.192.0/19

74.207.224.0 - 74.207.255.255
74.207.224.0/19

97.107.128.0 - 97.107.143.255
97.107.128.0/20

173.255.192.0 - 173.255.255.255
173.255.192.0/18

178.79.128.0 - 178.79.135.255
178.79.128.0/21

What we need is a nice big list of IPs and IP range lists to pin to the top of the forum for everyone to share.

Problem is I can't use hard firewall blocks otherwise I'd never figure out where these crawlers are using the data.

Someone has to make the sacrifice! ;)

But it [big list of IPs] would likely end up just like the big lists of UAs that used to get posted a couple years ago. Trouble is, what's bad for one web site is welcomed at another.

Example: Many of us block all the social parasites (twitter bots, linkedin, facebook, et al) but some webmasters thrive on their data being syndicated with the help of these harvesters.

[edited by: keyplyr at 7:33 am (utc) on Nov 6, 2011]

Example: Many of us block all the social parasites (twitter bots, linkedin, facebook, et al) but some webmasters thrive on their data being syndicated with the help of these harvesters.

I really don't have a problem with how people use the list either way.

My issue is validation, as long as it can be validated, it's better for everyone.

Those user agent lists without IP context are useless, and those were blacklists, again useless IMO unless you use them to build whitelists! hehe.

FWIW, I actually let facebook into my site because their security system does validate links are OK for facebook users to visit, which IMO is a good thing.

My linode list is:

66.228.32.0-66.228.63.255
69.164.192.0-69.164.223.255
72.14.176.0-72.14.191.255
74.207.224.0-74.207.255.255
96.126.96.0-96.126.127.255
97.107.128.0-97.107.143.255
109.74.192.0-109.74.207.255 (GB/UK)
173.230.128.0-173.230.159.255
173.255.192.0-173.255.255.255
178.79.128.0-178.79.191.255 (GB/UK)

Bill - I have a list of over 3800 "server" ranges of which probably 2500 or so are true server farms. Others are either invasive statics or me mis-interpreting the signs or even, in the case of RU and UA, applying a vindictiveness I probably shouldn't after a particuarly bad batch of /21 /22 /23 and /24 ranges.

I would offer the list but I feel it's too mis-leading and, in part, down to personal prejudice.