What does it matter how fast you go compared to rest of us.

Speed is relative.

It sounds like you can keep up with groups, and ride the distances you want. So, your speed is just fine.

For each person, how fast they want/need to go is different.

I average 15-16 mph on 20 mile to 80 mile rides. This pace can sometimes be slow as I get dropped by some groups. Whereas, it is just right for other groups I ride with.