I think there's a big difference between being sedentary in cold weather (when your body needs to generate heat to stay warm) vs. strenuous exertion (when it's still throwing off heat to avoid being overheated, just not as much as we do when it's warm).

It's probably a remnant of evolution that we're hungrier when it's cold - when we had nothing but our own and each other's body heat to keep us warm, we needed the extra calories.

I've sure experienced the "ravenous after swimming" thing.