To the OP, try to remember that you just can't generalize about what others do or what you should do.
There are plenty of super fit people who ride with significant medical "things," and I have found that you can't generalize about them, the same way you can't assume an overweight rider on a beater bike, wearing jeans will be slow... I've been dropped by those types of riders more than once. Quite a few people here have stuff that I think would surprise you. For example, my DH is riding quite nicely with 2 stents and can ride quite fast with people younger than him. His cardiac hx is genetic and it happened despite riding 4k miles a year and being thin. I have osteoporosis; again, nothing probably would have stopped it. I have every risk factor for it, my grandmother and mother had it and it does not stop me from much, except I did think it wise to give up mountain biking (I sucked at it anyway).
I guess what I am trying to say is that the stats you gave are pretty good for a new rider. If you want to improve, than you've got a good base. But, don't be too hard on yourself. I gave up comparing myself to others long ago. It took me a long time to find a group to ride with. They are almost all over 45 and the oldest person I rode with in that group is closing in on 90. He led a ride when he was 85! The group has a variety of people,most of whom have been active for years. A good contingent have run the Boston Marathon and do the PMC. Some are slower now and some are still riding incredibly long miles at a pretty fast speed. It took me a long time to find a group that I was compatible with. On days I feel great, I go out with the "spirited" riders. Other days, I stay with the main group, even though I know I will be riding a bit slower than usual.
Just be kind to yourself. Sure, my "regular" friends think I am crazy and can't even imagine what I do and that's what keeps it all in perspective.