According to a recent Singletracks post on facebook, guys supposedly LOVE IT when women pass them on the trails. As long as they're hot.

Ughhhh...the whole thing drives me nuts. But I'd personally rather that no one notice I was a female when I was riding. And I think in your situation the guys are being a little sensitive.