I wish I could remember where I read the article, but recently there was a great article that quoted a 10 year old medical study on exercising when ill. The gist of the study was that people who exercised during a cold didn't get sicker than those who took off. The colds for both the exercise group and the non-exercise group lasted the same amount of time, but the exercises were more likely to report feeling less lousy.
That justifies my exercising with a cold. However, like others have said, I do tend to take it easier and prefer indoor vs. outdoor riding for the control over the workout.
ETA: Found it http://www.nytimes.com/2008/12/25/he...best.html?_r=1