Same here. I could have never done what I do now on my bike in my 20's, unless I trained. I did nothing athletic until I was 24 and 25 pounds overweight. When I did ride, it was slowly and never more than 10 miles. I do need more recovery time as every year passes, but I am much tougher mentally. I just finished a 4 day tour of 150 miles, with each day having at least 2 climbs of 10-15%. None of it was flat. Now, I might not be as fast as others, but I just keep thinking that there's only about 5% or less of the women my age who do what I do.
Keep working at it!