It's my understand that drafting without asking is rude. I haven't had it happen to me but I would probably treat if like a tailgater - if he won't get off your wheel, then just pull over and stop. Take a break, eat something, catch your breathe - let him go on by. It isn't worth it to ruin your legs becuase of him - coz I'm guessing by the time he finished His ride He wasn't burned out!
Life should NOT be a journey to the grave with the intention of arriving safely in an attractive and well preserved body, but rather to skid in sideways, champagne in one hand, strawberries in the other, body thoroughly used up, totally worn out and screaming: "Yeah Baby! What a Ride!"