I am very lucky, I work downhill from my home. So I rarely break a sweat on my way in. However, depending on the season,I carry rain pants and a rain coat in my Ortlieb panniers. They are not cheap. But they do the job.
I used to be very worried about the weight of them but I don't even notice it as I scale my hills on my home commute, even when it's HOT