The bike club I've been riding with recommends that you start out riding with a group one level below what you really think you are. Would that maybe help you find your comfort level in group riding, and then move up in a little bit to match your real riding ability?
(unfortunately for me, that lower level would be riding our bikes around the block, and getting popsicles from one of the cool moms)