Sounds like Baboo is on the right track - if you want to go faster or ride longer it just means a bit more training. I have time constraints too - sometime I just want to relax on the couch after work! I try to train in the morning but that means short rides for me.
I am turning 50 soon too! I bought myself a bike 3 years ago and do most of my riding in the spring and fall. I do average 4 days a week at the gym throughout the school yer (which includes 2 really good cardio days). This is not a lot of training but I am able to ride anywhere from 13 to 15 mph average without a ton of bike training and without feeling like I'm struggling to keep up. I just finished my 1st century this Sunday too. I did the Montauk Century on Long Island (near NYC) - it is mostly flat and I did it with no problem. I'm hoping to do more bike training this summer. I want to celebrate my 50th by doing a more challenging ride in the fall.