Ha ha, story of my life.

Depending on the distance and difficulty, that 5 minute gap can be a LARGE difference in speed. So chances are you're not going to get faster just by tailing around after the fast group - if you tried to do speedwork from off the back, you'd probably wear yourself out before the ride was over.

Can you ask two or three of the slower members of the fast group, who might be closer to your level, to get together with you to do speedwork on days the club doesn't ride? That would be the easiest way for me to find motivation.