Well, I don't think 650's slows a person down.

I have a Cervelo with 650's (a tri bike). I can easily hold a 16 mph average into a strong headwind and on flats. The speed I believe is from the aero position.

Whereas, on my roadie, I would typically hold a 14 mph average into a headwind.

The other weekend I took the Cervelo out for a 20 mile ride and averaged 18.5 mph.

This past weekend I took the Specialized roadie out for a 60 mile ride- with hills- and averaged 14.5.

What am I saying? Basically I'm saying that I don't think the wheel size makes me slow or fast. The bike, the riding situation I am in, makes or breaks my speed. I have certainly done higher averages on my 650's than my 700's (roadie).

Otherwise, if you get 650's, always carry spare tubes, because it's unlikely anyone else will have that size.