No, you are not a whimp.
It just sounds like you need to find a comfortable cruising speed on your bike. One where you aren't working too hard, and with the proper rest stops and food intake... you could go for hours.
When you hit the wall... what does it feel like? What is happening? Is your breathing labored? Do you lose all energy? What have eaten up until that point? What did you eat before the ride? Have you had any rest stops? Is your average higher or lower than when you go, say, 20 miles on your bike?
There is no reason why you can't go any further on your bike. Answer a few of those questions and I bet we can figure out why you are hitting the wall when you do.
"Life is not a journey to the grave with the intention of arriving safely in a pretty and well preserved body, but rather, to skid in broadside thoroughly used-up, totally worn out, and loudly proclaiming: WOW WHAT A RIDE!!!!"