I can see it in a couple of ways. Sometimes paths are so crowded that it can be nearly impossible to pass safely so you end up hanging out behind someone waiting. Really you should try to back off so that you aren't breathing right down someones neck in this situation, but everyone has their own personal space and some need more than others to feel comfortable.

Sometimes I end up behind other people because I'm supposed to be out doing a zone 1 recovery ride and I know that if I pass this person/group that I'll end up going much faster than I want. I try to hang out at a respectable distance and often someone in that group will drop back and chat. At 5 feet and female I'm not exactly a picture of threatening either though.....

Now when you are clipping along on a quiet trail or road and you pass someone who then latches on to your rear wheel and sits there saying nothing.... rude - or sometimes just clueless. Just try to be sure when you pass someone you do it decisively. If you pass and slow down the person you passed will often unconsciously speed up a little and next thing they know they are drafting you, whether they know how or not. Of course there are always those guys whose egos won't allow little ole 5 foot me to show them up and try to involve me in a silent "race". Those are the worst ones. Get off my tail dude and if you have to pass like I said do it decisively don't pass me then slow down - that's just annoying and now suddenly I'm drafting you and I don't want to be. (of course I kind of think its fun when some guy blows by me, but can't maintain that speed for more than a little bit so past I go again in a mile or so. I usually don't see those ones again since they burned it all up to pass me the first time)