How long does it take you to ride 20 miles? How many ozs of fluid are you taking in?

I would guess you need to eat and drink more. I'm a big advocate of protein before riding. Otherwise, you need more for breakfast. I would also plan a snack stop, instead of just eating if you feel hungry. Sometimes, I get distracted and forget to eat--then I'm ravenous. I also love a post-ride smoothie.

Dehydration is a common cause of fatigue.