For me, speed just increases a little bit every year. I ride 3-4 times a week and do spinning classes. I am 58 years old, so I don't expect huge jumps in fitness, but I've been happy to have improved speed every year. As you gain fitness, muscle, and form you will get faster. I am a slug on hills but have gotten a little better every year on that too.

Just keep riding! And riding with a group can be a lot of fun. You learn from them, plus you benefit from the power of the peloton.