Two thoughts come to mind immediately.
First, you might not be allowing enough time in between rides to allow for recovery. I struggle with this one. You just can't push it every single day without allowing time for your muscles to rebuild.
Second, it doesn't sound like you're eating enough immediately after your rides. I do a recovery drink immediately, but I still need food - a real meal, with some protein. Otherwise, I'm hungry and run down for the rest of the day.
For 3 days, I get to part of a thousand other journeys.