We've talked about this here before (although I can't remember where), but it's quite a debatable topic. I'm 6 inches taller than you and have raced on 700's and 650's. I'm now racing on my TT bike on 650's and I LOVE them far more than my 700's. For me, acceleration was MUCH faster on the smaller tire, and I feel like I'm much more proportionate on my smaller bike and wheels (aero and speed wise). Some people will argue that you have to pedal more with the smaller wheels, but it has not affected me one bit. Maybe I pedal a little more, but the power I can generate far outweighs the little more spin my legs have to do (and they can use all the spin they can get).
My advice is to go with the 650's. Just my $.02