I know how you feel. I've felt as if I were holding others back and have ended up riding alone. The last time out our group leader had us cross a dual highway. I balked at this as this highway is notorious for auto accidents. He finally convinced me to cross but didn't wait for me! I ended up turning around and riding alone.

I'm not really a group person, I try to push myself by extending my mileage and adding hills. I'd ultimately like to find someone of similar ability but until then I guess I'll ride alone. If you like group rides I say don't let this stop you. Keep riding with the upper group, the best way to improve your skills is to surround your self with those who are better than you.