How long does it take you to ride 20 miles? How many ozs of fluid are you taking in?
I would guess you need to eat and drink more. I'm a big advocate of protein before riding. Otherwise, you need more for breakfast. I would also plan a snack stop, instead of just eating if you feel hungry. Sometimes, I get distracted and forget to eat--then I'm ravenous. I also love a post-ride smoothie.
Dehydration is a common cause of fatigue.
"Well-behaved women seldom make history." --Laurel Thatcher Ulrich
'09 Trek WSD 2.1 with a Brooks B-68 saddle
'11 Trek WSD Madone 5.2 with Brooks B-17