I would think that at least on the components side, you'd have to have Alaska winter-type air temperatures--or even colder--to make them brittle, though I'm not exactly up on metallurgy. One thing you do have to worry about with winter riding is the road salt, so make sure the bike gets a regular bath after riding.

I think it comes down to your personal "lower limit" and proper clothing. I don't have a hard time keeping my core adequately warm (layers, baby!), but I do have issues with cold feet, hands and face. I'm working on solving the hand problem, and I have a balaclava that needs surgery so I can put a ponytail through it. My limit is about the freezing point, and I won't ride if it's wet. (I largely ride on paved trails, so no salt for me to worry about.)