What puts me off riding..

[jezbael]

Many of the same things that you mentioned: wind, rain, weather. Living in Canada, I'll also add cold. I won't ride in the winter, because I don't think it's safe. But even in the spring, after the snow is gone, I avoid riding until it's warmer. Also, thinking about it as exercise rather than something fun to do.

[Crankin]

Stop signs or lights that end in an uphill.
Seriously, after ten years, I still cannot maneuver this. Even a teeny, tiny uphill, I cannot just push off and go. You can give me an 18% grade to climb instead and I would be happier.
I know what to do. I just can't do it. There have been quite a few embarrassing moments because of this, which entail scooting across intersections with the unclipped foot, getting caught on the saddle with my shorts, and the best one, actually getting going, but my foot that is unclipped gets in a weird position, the pedal is in the up position and I almost crash while I am riding with one leg.
All of my rides are planned around these types of intersections.