First of all, what's your body fat % (by a reliable method)? (You don't need to tell me , just think about it.) Do you really have those 3-5# to lose, while keeping enough fat stores to give you endurance and good hormone balance?

For me, it's definitely possible to lose weight while increasing distance. In fact, long easy distance is the most reliable way I know to lose weight. Try making one of your rides longer (75-80 miles) but at a relaxed pace. When I was young, I could increase speed and strength and lose weight at the same time, but I kind of doubt that's true now. (I purposely haven't tried to increase my cycling performance this time around; I know I've gotten stronger and faster since I've come back to cycling, but it's been very gradual.)

The most important thing for me is eating immediately after a long ride, when I'm not hungry at all and really have to force myself to eat. If I don't, I'll be ravenous for the next two days. Also, I need to replace calories during any ride (or run) longer than an hour. That never used to be true when I was younger, but it sure is now. Just 100-200 calories an hour, plus some protein if I'm going longer than three hours. That still gives me a marked calorie deficit if I was aiming for weight loss, but helps my performance enormously.