I just built a custom frame with 650c wheels. When I demoed bikes the only one I could find small enough in the brand that I wanted was a 48 cm on 700c wheels. The bike was too big for me, which I noticed, but I couldn't tell the difference between wheel sizes.

If you're worried about a harsher ride run less air in your tires, I run about 90 lbs. I also got handbuilt wheels for my new bike. They were designed for a small 105 lb rider rather than a 180 lb rider. They are more compliant than stock wheels and give me a very smooth ride. I read somewhere that a 650c wheelset will be 25% stiffer than a 700c wheelset of the same build. The builder was in Santa Fe, New Mexico and he has access to a company that makes rims for a light rider.

I have no issues with keeping up with others. Since I got my new bike my friends call me little lightning because I often catch up with them and disappear ahead of them.