I think that you have to decide which is more important to you (now, now, be sensible). It is not his fault that he is jealous, you would also be jealous if he rode off with a woman and left you behind. I suggest you find a female riding partner. It may be hard to find a riding buddy, but is is even harder to find a good man!!!
You never know if you tell you husband "you are right dear, I won't ride with so and so cause I love you and respect you", you never know, he may feel so bad about pulling you away from riding that that he will insist on you riding with him.
Good luck, and remember, good men are hard to find!