Does Bicycle and body height affect how fast you can ride?

[Crankin]

At least half of the people in my Wednesday group are 70+. They ride 3-10K a year and some of them also run, hike, etc.
Some have slowed down, but others, in the faster half of this group are at least 50-65 years old. I can't keep up with them most of the time. They invited me to join their other "invite only" group. I went once, and did a 50 mile ride on the first 80 degree day of May 2015. I sincerely thought I would die. Was happy there were two physicians riding with me. I didn't let on that I was suffering, but told them to go ahead on the last climb. They are super nice guys (I find the women who ride with them not so nice). These people are all my role models. About 10 years ago, one of our rides was led by a guy who at the time, was my dad's age. He is now 90 and still riding, after almost dying a few years ago.