This happens to me too. I feel sluggish and don't want to ride, and usually around mile 4 or 5 my left knee starts to complain. I used to stop and have water at that point, and sometimes I still do, but I've learned that if I just keep going I'll feel great. This happens to me even when I've warmed up with some yoga or running around the house a bit. I have the same thing in my yoga practice - those 5 As and 5 Bs (sun salutations) that I always start my home practice with are the hardest part of the practice, and everything after feels great. I guess I've just gotten used to it.