Well Google maybe should have the raters affect the SERPS to a certain degree.
Again, this is not what the raters do. They rate algorithms via double-blind samples of search results and tests. Raters in fact do not directly affect rankings at all.
By down-rating an algorithm that allows a lot of spam sites to rank well, though, raters might indirectly be down ranking spam as well, but it's not a a question of a rater downranking a specific spam site. Such a system could never scale, and would be much too slow to react, say, to push-button spam.
There is, btw, a several page section in the last raters guidelines I checked out about spam as described by the guidelines (and described pretty well, I think)... , and spam sites do get the lowest of the low quality scores.
I should mention that there is a team of human "reviewers"... not quality raters... who do check out and review spam sites directly, and that is the Spam Team. Google allows multiple ways for its users to report spam... and the spam team is purported to have great insight into individual sites remotely. Again, that's for identifying spam... not for ranking billions of pages on a competitve basis.
Much spam relies hijacking of sites or domains... which involves the exploitation of vulnerabilities often caused by poor site maintenace and unpatched code. As Matt Cutts used to discuss at length, Google cannot patch the vulnerabilites in code for webmasters. Webmasters have got to do that on their own.
One further thought.... If raters could directly affect the rankings of a site, that would open up the system to a lot a problems. Can you imagine the outcry, say, if it was determined that a particular rater could make a site go up or down in the serps?