What Smiling Cat said, is what I was speaking of. While I don't want to puke, doing intervals 1-2X a week will definitely help you. And, I have found winter training either on the trainer or in spin class allows you to do this is a way that sometimes doesn't happen outside. And, make sure you do winter sports that are aerobic!
It is also hard for me to find a group to ride with. I have one, but now they have split into 2 on a lot of rides. One group, I am one of the stronger ones, and it makes me feel good. But, I never get better if I only ride with them. The other, I can keep up, but not on a 45 mile ride with big climbs. They make me feel badly, despite the fact they are very nice. My goal this year, is to ride with the faster ones in the winter, or in the early season, when they are milder in their style. But, I often get sick of trying, and I end up just settling back into my regular routine.
Yes, there will always be someone faster. A few years ago (when i was a faster rider), I got dropped by a guy wearing jeans, a backpack, and on a mountain bike.