Alright, this is gonna sound harsh but here goes.

You are being way too girly about this. You don't enjoy the group rides because of the drama, and because you " want to go as hard as I can and not worry about someone's feelings being hurt."

The group rides aren't working for you. So stop going on them, and go out for beer with your friend instead. Women are never going to get where we should be in society and sports and everything else if we keep apologizing and worrying about somebody's feelings being hurt because we are good at something.

You don't have to be a jerk about it. Don't say "you're a slug and a drama queen and immature", even if it is somewhat true. We all fall short of perfection in many areas.

Above all, stop the analyzing and justifying. That's another girly tactic we use when we want to do something that we really enjoy. It gives you a thrill to ride fast and train hard, so do it and don't apologize.