Okay, I'm going to just toss in that it depends on the group. SOunds like DH's group isn't the right one.

If you're afraid you'll feel alone, connect with somebody or accept that you'll feel alone (it probably isn't fatal for one bike ride). I'd say "if it isn't for you, then don't go back" except that even the same weekly group ride can be very different on different nights, so I would psych myself to try it, say, three times before making a decision.