Im used to riding men's frames, as fixed crit bikes are almost never women's specific. Being on the short side (5'6") for men's bike sizes, I use a shorter stem and have my saddle a bit further forward, and I've never had an issue. But those are only for two to three hour rides, so I wasn't sure if I should look into women's specific bikes for longer rides.
I have mostly been looking into Specialized as they are the more common brand in the area, or at least the stores have better stock of. Trek and Cannondale are really the only other accessible brands, but I'm less familiar with their bikes.