I got faster not long after I started riding again as an adult by doing a couple shorter (20-25 miles) rides with a faster group of riders. At first, they kind of sucked, but as those shorter rides got easier, I started going faster bit by bit on my longer weekend rides. But I will tell you, you have to keep training that way to keep your speeds up, so if your heart isn't in it, it's okay to just keep riding as you are. There's nothing that says you have to go faster (although it's also a perfectly reasonable goal).
Part of getting faster is learning good technique and finding a cadence and choosing gears that allow you to maximize your aerobic capacity and leg strength.