It sounds like you are doing really excellent to me! If you keep hanging with faster people like you did, even for a short time, you will get faster (though you already have a respectable speed going). I was out riding with the trailer/road bike last night when a guy on a hybrid pulled up alongside. We got to talking and he said he's been able to average 18mph. You've already proven you can do that kind of speed, it's just a matter of time and challenging yourself before you average that, with small sprints upwards from there. I don't think hybrids are terribly slow, especially if you pay attention to air pressure. Sure, you have a little bit more resistance and you are more upright than a road bike so you have to work harder but even a mountain bike with knobbies can do a 20 avg over 20 miles on the road. Depending, of course, on the road!
I used to have an open mind but my brains kept falling out.