I've done the same thing a number of times. A few years ago, I had a bike built up from scratch. I never felt very fast on it. A couple of years later, I moved some of the components from that bike over to my other road bike and, again, I started feeling really slow. Finally, we realized that the rear break wasn't staying centered. It didn't even take a topple or anything to get it wonky. It would just spontaneously move. We fixed it, but not before I had a number of rides that just kind of sucked.