Rain, wind, temps below freezing or above 90-something, depending on the humidity. In summer, I usually ride early in the morning (have to be out by 8, back by 10 or 11), though there were plenty of weekdays when I rode during the day. I'd get out at 10:30 or 11 and come home at 2. How I did it without frying my tiny brain, I don't know.
The wind especially this year because my new "home" trail isn't nearly as protected from the wind as the one I rode last summer, and even more so now because if I push too hard a gear (as I tend to do in crosswinds in order to feel stable), the abdominal muscle I injured back in November throws a fit.