I have those rides, and I've come to realize it's usually not me, but an outside influence, like the wind (this may be a trick to make the hard rides better ). If I'm really slow, it's either a headwind, or my brake is rubbing.

How you feel on hills is a better gauge to how you're really progressing. You're feeling better on the hills, you're getting stronger for sure!