I'm 40 (but had a complete hyster, so I'm more like 50!), and I was never athletic before I started "cycling" about 4 years ago. Can't really call it "cycling" - it was just getting on my beach cruiser and pedaling once in a while for fun.
When I decided to start using it to try to get fit, instead of just for fun I always felt like I had a little person on my shoulder saying "look at you, you loser! Can't you go any faster than that? This isn't a very big hill at all - you're such a weakling!"
One day, I decided to stand up to that little voice. I said "I'm having fun, and as long as the bike is going fast enough to keep upright, I'm doing alright." Many times going up a long hill I had to just keep saying "I'm upright. I'm upright", like a mantra. That's what kept me going.
One nice day when I didn't have any committments I told myself I had all day to do it, and I was going to go 15 miles if it took me the whole day. I filled up a couple of water bottles and crawled along for 15 miles. It took me almost 2 hours but I kept the bike upright. I felt so accomplished - 15 miles was like a marathon to me. Then I decided to ride 30 miles. It took me 3 hours.
Finally, I found this forum and some nice ladies said "30 miles on a beach cruiser? Seriously? You should get a road bike" and I was like "nah, I'm not good enough for a road bike," but they talked me into it. Last year I did 65 miles one day - it took me almost 5 hours, but I did it!
I am just now seriously thinking about putting some real effort into getting faster. I have no idea how to do it because, like I said, I've never been an athlete. I was always picked last in gym class - AFTER the fat kids and the kids with glasses and inhalers.
Maybe it's easier for me because I don't have anything to look back at and compare myself to (except that when I was younger I had a very nice body). It's so exciting to me to do something "sporty" and makes me so proud of myself.
You have your old self to compare your new self to so it's probably harder for you - but as long as you're keeping the bike upright, you're moving forward.