I do not think that weight, alone, makes a difference. I have seen women who would be considered overweight who regularly cycle who could kick my butt any day on a bike (I'm 5'6" & 137 lbs.). Yes, it does require more strength and endurance to carry around more weight, but weight is not the only factor. It sounds, too, like you could benefit from continued strengthening overall. Eventually you could build both the strength and stamina to go longer and farther. I know I can go a lot farther on flat ground vs hills.
Have you tried riding more than 5 miles on completely flat ground to see how it goes?