Just my personal opinion:
I think you should talk to your husband about it. It does not set a healthy precedent to have your husband be in a role where he can just eliminate your male friends for little reason other than his own insecurity. I have friends of both sexes, and so does my husband. Some of our friends are shared friends, and some are just personal friends. Having lots of friends is healthy and good. Riding with others is safer, too.
I don't believe we should only be able to have friends of our own sex- gee that eliminates half the wonderful people on earth!