It is rude and dangerous. If you don't know you have someone behind you you can make sudden manoeuvers that can endanger that person and eventually yourself.

It happened to me once, I slowed down and the guy slowed down, I motioned him to pass me and he said he didn't want to. I'm about 5'4", the guy was almost a foot taller than me and a good 50 pounds heavier, he would have been great to draft from. I really wasn't impressed. I eventually stopped at the side and he stopped too, I told him to pass me because it was really rude and no I didn't want to ride with him and draft him. He acted as if he had no idea why I said it was rude, it seemed to be something he did all the time. Maybe indeed he didn't know. Anyway, he took off, I ate a bar to calm myself down, and took off again, riding slowly to avoid seeing him again...

I think it's important to voice your opposition, otherwise how can people learn?