I understand how you feel. I ride with a pretty fast club, and can hang on for about 20 miles, so then I just drop off. I've taken the philosophy that once it becomes un-fun I slow down. I'd rather be dropped and finish on my own or with a couple other slower riders than be in immense pain. I can say however that pushing myself has helped increase my speed. While I still get dropped, I can hang on longer. So it's not all for naught. What did Greg Lemond say about it not getting easier, you just get faster? I have another friend I ride with who's a total hammerhead. We ride together for a while then he gets the itch to take off so I just think "there he goes" and keep going by myself. Happened yesterday during a century ride. I can say he's in a lot worse shape than I am today though
hang in there
"Only the meek get pinched, the bold survive"