I definitely wouldn't do the hard workout on a recovery day if fatigued. You can become injured. I would do the recovery day... then on the days when you think you will miss a workout, try to squeeze in at least a 30 minute workout. I have a treadmill at home, so on those awful busy days, rather than skip, I look for 20-30 mins on the treadmill.