yes, that's part of it. You don't draft off someone without their permission.
the fact is; you've been practicing with a certain group of people, and you know them. Enter a stranger; and start pacing with them. Do they pay attention? are they steady? Are they going to bump your tire and make you crash? You just don't know, and no one knows about your riding skills either. So most people DON'T do it with folks they don't know, at least not without permission; and then permission is not always granted.