I more or less did what Crankin did. My standard weekday rides ranged from about 15 to 25 miles. I used the weekends to build up to longer and longer rides. There were a couple of plateaus--30 and 45 miles if I remember that ere challenging. From there, adding miles got easier.

Speed work came a bit later. While I did get faster by simply riding, I didn't get appreciably faster until I started riding with people who pushed me.

I remember being really nervous to finally do a group CIBA ride, Catrin. Once I finally did, I realized they were far easier to do than I thought. If you're worried about speed, do a weekend ride rather than a dinner ride. There's no set pace for a weekend ride.