How often are you riding? If it's only once every couple of weeks, then it might just be general fatigue from the road bike position (and possibly a more jarring ride). I had that problem last summer when I only rode occasionally - each ride wiped me out with aches, pains and heavy fatigue for the rest of the day. This year, I ride more frequently and I don't have the same problem.

I agree with letting a little air out of your tires. If you currently pump them to the max recommended value (like I do), you will probably see a great improvement in road feel if you back them off to the middle or lower end of the rated range for your tires.