i don't think it's your sex, I think it's what physical condition you are in. Even so, some people are mashers (like your hubby) and some people are spinners. You sound like a spinner. I know some spinners who ride hundreds of miles at a time.
and yes, I think some men have trouble understanding how anyone could be different from THEM. You keep going the way you are and you will get stronger, and you will be able to use stronger gears, but maybe you won't want to!
keep spinning!