Yea, I think this is one of those fundamental gender differences that neither of you should try to change - just accept!

You might try going out five minutes *before* he does. That way you get to be leisurely and he gets the challenge of catching you.