I've spent quite a bit of time evaluating page-load speeds of my website's pages.

The websites I've used for testing (the same big web-based services everybody uses) give you:
1) A ranking, like 90 out of 100 (meaning that your optimization isn't quite perfect yet).
2) They also show a "waterfall" graph showing the timing of the events as the page loads.

When I look at their recommendations on how to improve performance, 95% of the suggestions (and all of the more major ones) have to do with Adsense (.js calls, redirects, etc...). So basically there's very little I can do on this front.

When I look at the waterfall graph, 1/2 to 2/3rds of the download time for my pages is, once again, tied up with adsense functions.

Have I overlooked something. Is there any remedy at all?
I really don't see how even an all-text page with 3 adsense blocks and absolutely no calls out to anything else except Analytics could every be more that a mediocre performer in Google's eyes.

As far as page loading goes, my content loads first (so the visitor does have something to read immediately), the ads then appear moments later.

I personally would consider this an ok visitor experience, yet since my pages' performance are dragged down by Adsense, Google Anaylitcs shows me data that says I have a slow website (and therefore, in at least theory, am penalized in the SERPS).

Is this simply the penalty of using Adsense and there is no solution? Is anyone having a different experience?