This is why I choose the groups I ride with very carefully. I am riding for fitness, but also for FUN. I always seem to be between groups in terms of ability. When I ride by myself or with my husband, my average is between 15.5 and 16.5 if it's not too hilly. However, there are always hills where I ride. I can't get home without riding up 2 pretty big climbs. I can sustain 18-22 for awhile, if it's really flat, but like I said, no place here is really flat.
Riding with my husband forced me to improve for the first few years. Now he is happy to ride at my speed and gets his faster rides when he commutes to work. I am too fast for the "slow" groups and when I have gone on rides that advertise a 15-16 average, it always feels fast to me, even though it doesn't feel fast when I am alone. However, I find that I can out climb a lot of the riders on these rides; I am a downhill weenie and that brings my average down.
I don't do well when I am constantly trying to keep up with others; it doesn't make me stronger, it just makes me angry and feel like I am weak! I can compete with one person, but a whole group, no.
Of course, I have to make myself remember that most "regular" people think that what I do is crazy and that I am fitter at 54 than I was 10 years ago.