Just enjoy your ride and YES there are plenty of people out there who will be going faster or much much faster than you. There's always a bigger fish in the pond than you. Besides, its not a race. The riders who do pass you, are they older or the same age as you? Probably younger. 30 years ago, I was so much faster than today; but, I realize that it is unreasonable to expect to perform as well today as I did 30 years ago. I'm not as strong, not as fast, don't have the endurance and I take a lot longer to recover. Its life. sigh...

Remember too that disgraced Lance Armstrong quipped that after his bout with cancer, he went out on a ride only to be passed by some old woman.

bigger wheel, you need more torque but less rotational speed. Smaller wheel, you need less torque but more rotational speed. And power is torque multiplied by rotational speed so for same traveling speed, it turns out to be that power needed is same whether its big wheel or small wheel. And this is where thing called gear inches come into play. Basically if you have a smaller wheel, you need to be geared higher than on a big wheel. Even though you may be geared really high on smaller wheel, it basically takes the same amount of effort as it does on a big wheel geared low.

Another reference point, racers and serious weekend warriors put in 150 to 200 miles a week minimum. They would also include interval training and its not pleasant nor is it pretty (chucking biscuit is pretty common).

Agree with what everyone else has said already.

Enjoy riding while you still can.