Yesterday was close to a record for me with 7 degrees (F) and 20 mph winds. Four is my personal best... but both times, I wasn't heading towards hypothermia... once past the three minute chill at the beginning, I warmed up and yes, it was fun. I don't mean "It's fun because when it's over I can tell a story about it," either; fun in exactly the same way riding when it's twenty degrees warmer is fun.
Could be the Goretex, though I'm very grateful to the fella who decided he didn't need it when he retired and gave it to me!
(AMAZING to ride on ponds!!!)