I am going through the same thing. I think age, uh, has something to do with it. I feel like I have one or two good fast days every two week at this point. I have been riding for almost seven years...
I didn't ride for a week because last Tuesday I felt sick from alllergies during my commute. Took Wed. off. Did an intense yoga class and weights on Thursday. Rested Friday. Saturday, it rained, did another hard yoga class, with a 10 minute cardio warm up. Sunday i went on a 3.5 mile hike (easy).
Rested yeterday. This morning my legs still felt like lead. Two weeks ago I got to work in 56 minutes. Today was 59.20.
I always take one full day off, sometimes two. I ride with a slow friend once a week. Try to stretch every day and do yoga and weights twice a week. I feel like a wimp compared to others. My mileage isn't even that high!