Many of the same things that you mentioned: wind, rain, weather. Living in Canada, I'll also add cold. I won't ride in the winter, because I don't think it's safe. But even in the spring, after the snow is gone, I avoid riding until it's warmer. Also, thinking about it as exercise rather than something fun to do.