People of all shapes and sizes can get fast with proper training and consistency. A lot of the bike club folks ride all the time and work on speed with intervals and pacelines. Riding the same speed all the time doesn't tend to make one fast, but intervals help. I personally hate intervals, and as a result, I never get much faster, just a little over time, when I am doing a lot of riding.
Back when I was at my peak of riding and did many club rides (2003-2004, before I had a serious cycling accident in spring of '05), I gradually worked my way up to riding with some of the faster (B-paced) ricers -- but not the fastest club riders, which was fine with me. I was riding consistently and usually with people who were a bit faster than me, so just trying to keep up with them helped my speed over time, as did lots of training miles and quite a few hills where we lived.
Now, I only ride with DH or alone, and I am a lot slower than I was back in my peak, though I can still put in as many miles. But my typical average speed of 13-14 mph (sometimes a little slower or faster depending on which bike I am on, the terrain, number of stops, etc.) puts me in that "middle" area where the fast riders (like my DH) easily leave me eating their dust, but I am constantly passing people on cruiser bikes, mountain bikes (on pavement), families, etc. on my bike path rides -- the more casual, infrequent cyclist types or those riding very heavy non-road bikes, in other words.
I don't know why it is so hard to find people riding in the "middle" speeds, but it does seem to be.
Emily
2011 Jamis Dakar XC "Toto" - Selle Italia Ldy Gel Flow
2007 Trek Pilot 5.0 WSD "Gloria" - Selle Italia Diva Gel Flow
2004 Bike Friday Petite Pocket Crusoe - Selle Italia Diva Gel Flow