I would be honest and simply say that you have limited time for cycling, and that you need to make the most of it in terms of training and for you that includes riding as fast as you can; leisurely rides are not part of it. If they take offense, so be it. Most reasonable people would understand.