Well, I can relate to the feeling of being stuck in the middle. And it's true that to get faster you need to ride with faster people. But, I got faster (up to the point I am now) by riding alone or with my husband. He pushed me and I was able to get faster without all of the "stuff"of group rides.
I find that I am way faster than the "slow" group rides in my club and I lead intermediate level rides where we cruise on the flats at 15-18, but since we ride in hilly areas, the average is often between 13-14. There are always a couple of stronger climbers than me on rides I lead and I let them go ahead and wait for me at the top. But usually it's guys who really ARE much stronger than me. I have found one group that is exactly like me; the rides are social and I climb as well as all of the people who lead these rides. We go 15-17 on flats, and while I often can sustain 20-23 for awhile when I ride with my husband, I don't necessarily want to do this for a whole ride. Another group I ride with starts out really slow in the beginning of the season and builds up by the end. One ride with them, I had an average of 16.4, but it almost killed me. They don't warm up well and that puts me at a disadvantage. I got dropped at the end by a woman wearing sneakers and a t shirt
So, sometimes I am competitive, but mostly if I do that too much, it makes me start to feel like riding is a job. I don't want to ever feel like that. In fact, I often ride with people who are slower than me, to force me to slow down and smell the roses.