"pace creep" is what we call it and it's basically universal. Our Monday ride leader was completely surprised to find out that they'd totally and completely dropped a rider ... and he should have seen it happening and/or duly noted that when the group stopped halfway, that a small group was teetering in 'way behind. Nope. When we took off, they took ooooffff! again.
The ride leader is a really nice guy and a good ride leader, who basically got into the ride and forgot to check his speedo and didn't realize that what he thought was "the group" wasn't, in fact, the whole group. Our Monday rides get pretty big and we don't formally log in or anything. It *didn't* happen last week (though on the last five miles, when a faster group came around, most of the "slower" group joined the faster group).
It doesn't have to be a phobia. It's no fault of yours if the yingyangs lie and get on their adrenaline rolls. I would give it another try but be sure to tell the ride leader that you are reasonably sure that they'll be going much faster than their claims, based on your experience, so you'd like to know the route (and perhaps a shorter option).
Some people subscribe to the "hang on until you can't any more" and then drop off oh, right about the puking point. I refuse to get to the puking point.