No, 650c wheels will not make you slower. Your speed is dictated by the gear ratio. With a smaller sized wheel you need to select a different gear to have the same ratio, but the effort required to move at a certain speed will be virtually identical. It is true that smaller wheels have slightly more rolling resistance, but then they are also lighter which will balance things out. Furthermore, to build a frame in your size with 700c wheels will likely have either compromises to the handling, or toe clip overlap.