Well, I don't think 650's slows a person down.
I have a Cervelo with 650's (a tri bike). I can easily hold a 16 mph average into a strong headwind and on flats. The speed I believe is from the aero position.
Whereas, on my roadie, I would typically hold a 14 mph average into a headwind.
The other weekend I took the Cervelo out for a 20 mile ride and averaged 18.5 mph.
This past weekend I took the Specialized roadie out for a 60 mile ride- with hills- and averaged 14.5.
What am I saying? Basically I'm saying that I don't think the wheel size makes me slow or fast. The bike, the riding situation I am in, makes or breaks my speed. I have certainly done higher averages on my 650's than my 700's (roadie).
Otherwise, if you get 650's, always carry spare tubes, because it's unlikely anyone else will have that size.
"Life is not a journey to the grave with the intention of arriving safely in a pretty and well preserved body, but rather, to skid in broadside thoroughly used-up, totally worn out, and loudly proclaiming: WOW WHAT A RIDE!!!!"