Matt Cutts has already said this is a very minor part of the algo. I would not waste significant time worrying about.

Faster pages make your site more attractive to users and thus more likely they will spend more time on your site and will link back to you from their blog & twitter accounts. This is why speed is important to me.

I wonder how many adsense ads you are displaying, because 20 sec is too much

@goodroi
Currently speed is a very minor signal, yes. However that might change and could (will?) become more important.
Things like this [webmasterworld.com...] make me think when I read them.

My page loads in around 5-6 seconds but sometimes the loading never finishes (the page is long finished displaying though) due to flash Ads reloading stats scripts.

I display 3 AdSense Ad Units via DFP as well as 2 Units for a Wallpaper Ad

A single adsense ad can add 1.5 seconds to your page load times and cause 9 object calls as well as add inline css to your page. With adsense it's an all in proposition, you either use adsense or you don't at all, a single ad is nearly as bad as several.

A google search box and to a lesser extent analytics also draw a lot of resources, the little image in the search box especially.

So many javascript files Google, so many...

If it just would be 1.5 Seconds.

Like I said I had flash Ads constantly calling a tracking script and therefor the page never stopped loading which created page load times of 160 seconds. My newest record is 384.14 Seconds.

I don't want to deactivate rich media ads because the page loads fast, it just does not stop loading after it is rendered.

However I fear that some day Google will punish my website based on those measures

Someday they may discriminate against those of us who do not use their analytics ..( I know ..don't go giving them ideas ).. they sometimes appear to source their ideas and penalties, and even products, from the amphibious thought patterns of the proverbial "box of frogs" carrying modified bedbug genes.

But given the number of premium partners who also use rich media ads ( and the revenue that Google makes and splits with them ) I would not think that of itself it would make one lose brownie points with the Plex..

Unless, of course, If such an idea did strike them, it was only implemented against non premium partners, they are vague, mysterious, powerful, have hidden motives, and are quick to smite..which contributes to why they remind me of the Jesuits so much..with a little Bene Gesserit and the Guild added to the mix.

I think the real reason they registered Google.com was that CHOAM was already taken.

I just searched for CHOAM on google to see what it says now and the acronym result ranking #2 was great.Well done google. Doing a search on bing was a lot better.

I would not waste significant time worrying about.

...but what about it's connection to an adverse bounce rate and it's effect on rankings ?

Isn't this one of the common central concern's to usability interpretation issues in Panda ( yet to be proven )?

Re those tracking scripts - are you seeing that in all browsers? For example, I notice that some of my stuff is significantly slower in Firefox.

I found one Ad that was causing this with Chrome Network Analysis.
Never found another again but the insane loading times have to come from somewhere and when there is one Ad doing this there are most likely others.