I couldn't wear men's stuff, either. Heck, I can't even wear size small jerseys. Most of my jerseys are Terry, because they make all of their jerseys in x-small, no elastic, and they don't puff out over my small chest. I also like Voler jerseys, too. I like my jerseys tight.
There are other jerseys that fit me, but not with the regularity that Terry and Voler do. I couldn't wear the small sized, women specific jersey from Primal that I got from the tour I went on. It was huge in the boob area. I gave it to someone who needed it instead of the M she had ordered. They are going to send me another one and I will have it altered, since I like the print, etc.