I started riding again about three months ago after fifteen years of not riding. My first couple rides out, I was dying after a few miles (3-6). I can now do about 40, but my speed is pretty slow, especially if there's a lot of wind (I ride by the ocean and it gets windy). There are no real hills around here, but I can tell you that I'm huffing and puffing when I do the bridges, and I know they are nothing in comparison to what many people experience on their daily ride. It sounds like you're doing fine to me. I just started a two-day-a-week interval training class to help me get a little more fit. I'm signed up for two charity rides, and I want to increase my speed and endurance. I think it's just about getting out there and trying a bit more each time.
2013 Kirk Frameworks JK Special/Selle Anatomica
2012 Gunnar Sport/Brooks B17
2001 Calfee Tetra Pro/Selle Anatomica
1984 Raleigh Sport/Brooks B66