That's actually a pretty good average speed for a hilly area. It is very easy for your speed on steep or long uphills to drop down to around 3 mph... which is an average walking speed. It's also quite normal to need to get off and walk your bike. If you can't ride in a fairly straight line up a hill because you're breathing so hard and are wobbly, it's past time to get off.

It takes practice to get the full benefit of a downhill too. And sometimes it's not safe to try to get the maximum speed off a downhill. You might have bad pavement, or a busy road, or tricky curves... It is a judgment call. I don't try to go full speed down my nemesis hill. It has a cross street at the bottom with poor line of sight, pavement in bad condition, and a curve right after the intersection at the bottom. Not a good place for me to try to set a speed record .

Hills do really good things for your endurance and speed on the flat. So it's not all bad