I had a very similar fear to yours!

I'm 5'4" but I have quite short legs. My first bike was a 49cm men's model and it was way too big (but I didn't know any better). When it was time for a better fit, I was told by my LBS 'fit guy' that I needed a 44cm in the Specialized WSD I was admiring. I test rode a 51cm model and had plenty of standover clearance due to the compact frame, but the reach was too long. I kept thinking that a 47 would be better than a 44 because I have short legs but a long torso. I even brought this up when we were placing the order and he re-did the calculations. If I'd gotten the 47, he'd barely be able to make the reach right with the shortest stem available (I guess I have a long torso but short arms). With the 44, it was right as is. I agreed and ordered the 44 (I test rode another 44cm model in the store).

Now, well over 1000 miles later, I'm VERY glad I listened to him. We rode 75 miles this past Sunday and I felt NO pain. NONE. My bike really, truly fits me. Yes, I hate that I can't fit two water bottles on the frame and yes, I hate that the guys make fun of my tiny bike...but you better believe that I LOVE the fit. Love it! Moral of the story: don't be turned off by a small bike!