If you are having to tug pretty hard to get the tire to clear the brakes, then that is probably why you are knocking them off center.