I notice that people look over their shoulders when they are (I assume) uncomfortable with my drafting.
I was looking over my shoulder several times while he was back there, probably once every 20-30 seconds or so. If I know someone is back there drafting off me (and it's clear that they're there or I know them, etc.), I check less often and more to make sure they're still on versus "why are youthere?" kind of thing. I also kept checking after I dropped him and could see him back behind me. Eventually after the 2nd drop, he disappeared. I don't know if he stopped, or if the wind was too much for him on his own so he was going 2mph or what. But I was glad he was gone. I never felt afraid of him, the path was relatively crowded. Perhaps he didn't know the rules. That's certainly not a far stretch on the path.
"Only the meek get pinched, the bold survive"