Well, I don't think gender is the issue. My club recently had a "women's ride" where 2 riders were dropped, and no one even went back to look for them.
My advice would be to communicate that you want someone to hang with you in case you can't keep up. Also, he shouldn't have sprung a faster pace on you without warning you. That's just rude.