7rider, I have the same monkey on my back. I have got to get rid of the negativity. I am actually getting weird anxiety things before riding, although it doesn't stop me... some of it is my total lack of mechanical ability, still after almost 10 years of riding. I'm trying to give myself a dose of my own medicine: deep breathing and positive self talk.
I have always been too fast for the slow group and too slow for the fast group. Of course, this depends who you ride with, but it seems like I peaked in 2006 and it's been downhill since then.