ah, such a good question. I happily rode my hybrid for years, but had so much trouble keeping up with my husband and his friends that i finally caved and bought a used Bianchi. It was a beautiful fast bike, but it did not really fit me. So i put a bunch of money into it, trying to get a good fit. Meanwhile, the bike itself encouraged me to work harder (I cannot explain it) even though I was awkward and uncomfortable. I eventually got a custom bike, which fits me well and is a lot more uncomfortable. I don't ride in a real extreme position btw and you don't have to either. There is a bike, the dolce (Specialized?) which is not real expensive, but it has the geometry of some of the more expensive bikes. You might test ride one of those.
I had to make the leap myself. And you CAN go back, i still have my old hybrid.