I blocked all networks except Criteo (the others didn't earn anything anyway) in mid-March because it was annoying watching them being slow to load.

I just checked rpm before and after and it is completely unchanged after making the change i.e. no increase due to improved page speed and no decrease due to lack of variety in ads. I also checked analytics and that shows no change in page speed, but I'm not sure if that measures load time before or after ads are loading.

Criteo is often still showing as slow to load, which might expain why page speed hasn't changed, but does contribute a few % of income, so I'll leave it on for now but test turning it off when I get a chance.

edit: I should mention that over the years whenever I have changed server, hosting etc or made changes that effect page speed I have never actually seen an impact on earnings. The only thing that made a difference was adding media.net which definitely slowed down adsense loading and reduced overall earnings, so got removed