Beyond the possibility that your brakes are rubbing, my guess is that you just need to tweak your fit and position. It's possible that your seat is too low and/ or too far fore or aft. You'd be surprised what a few minor modifications can do. Hopefully, your bike fitting will help iron it out.