I hate to be a little disagreeable, but maybe you're eating too much. A 15-20 mile ride shouldn't take over 2 hours even at a 12/mph pace. Everything I've read says you don't need to worry about eating during a ride if it's less than 2 hours long. Is it just possible that you're having a sugar crash? Meaning you've eaten too many carbs and your body crashes coming down off all that sugar. Yes I said sugar because that's basically how your body treats carbs.
Just a thought. I may be right or I may be wrong.