I am no slower on my 650's than I was on my 700's. I find that I have a quicker acceleration on my 650's than my "big wheels."
I do think, however, that I'm slower on my 20" folding bike wheels. Maybe it's just that my little legs are turning more revolutions so they get more tired quickly.