Any relationship is only as strong as each member's willingness to provide edifying support to the other.
If this is a point of consternation in the relationship, then it is not edifying and will only have a negative impact.
Is this allowing the "insecure" one to dictate who your friends are? or Is it your willingness to submit yourself a discretionary relationship for one that is a true long term commitment? I believe the latter. I also do not believe that the fault is necessarily his 'insecurity' as you each have a stake in the strength of the relationship and a responsibility to nurture each other in your challenges.
I do not begrudge Silver having male friends that I know and trust.
I would have serious problem with her continuing a male friendship that I had problems with...regardless of the reason.
If you don't grow where you're planted, you'll never BLOOM - Will Rogers