Why not just tell him that you would rather ride at your own pace without the pressure that he brings to the ride? You don't have to ride with him if you don't want to.