It's not just the time or the mileage that affects how you feel after a ride. It's the elevation gain and/or wind. A flat 40 mile ride may make you less tired than a hilly 25 mile ride (and may take almost the same amount of time). My average speed never amounts to much because there are such steep hills that my odometer always says about 10-12 mph, even though on the flats, I'm going 13-17mph, and the downhills can reach 25mph. The up the mountain speed can be 4 - 6mph. It depends where you ride. The small grade hills are not a problem anymore. When I read about people doing all these century rides, I wonder how many are in places where the biggest hill is a freeway overpass. The farthest I have ridden so far is 49 miles. Nothing is flat. My legs are getting stronger.
I suggest picking a route with x amount of miles and seeing how long it takes you to do it. Then see if you get there in less time in the future. I have a nice 20 mile round trip that has alot of rolling hills (some steep) and no lights. I use it as a benchmark. The first time it took 1 hr. 15 min. to go out and 1 hr. to get back. It seems that the way the hills and wind are, it's faster on the back direction even though the same distance. Now I get there in 55 minutes, and back in 45 minutes.
The other thing that affects speed is your bike. Some bikes are just faster. In time, your conditioning is also better. Together your riding ability goes up. It really takes a lot of miles to get better. In fact, I can hardly believe that I can do what I do. Started riding at 56, and it's been a little over a year.