I agree with Spokewench in theory, however there's no way my ride is going to be the same if there is someone behind me that crashes because they hit my rear wheel. Being at the front gives me lots of responsibilities, and I am not always willing to assume those responsibilities. If I'm riding casually, it's one thing. If I'm training and going at high speed/high effort (as mentioned by the original poster), it's another story.
In Montreal on the F1 Grand Prix course road cyclists tend to get together on summer evenings and ride in very fast pacelines together (30 mph, sometimes more). It's relatively flat, but there are some sharp turns and some inclines that can bring cyclists to stand up, or pedals to hit the pavement, with catastrophic effects for all involved. One night when I was there there was a crash and one cyclist died at the scene. Not a nice way to finish an evening.
When I went there to train, and joined the pacelines, I knew that it was dangerous (and exhilarating!). I trusted some people but certainly not everyone, and especially not strangers. It worked the other way around: I would not have tried to join a paceline with people I didn't know. If someone I did not want in my paceline joined us, and we couldn't get rid of him/her (usually him) (at that kind of pace, there are no kind words or gentle educational methods, just orders barked over shoulders in heavy wind), I'd just peel off and finish the workout on my own.
I feel the same when I'm training on the open road, although of course the risk is much less if there are only two - not twenty - people involved.
I also agree with Lisa's point of view regarding possibly creepy strangers. Not everyone wants to have company on the road, especially not company that requires attention (signals etc. are attention).