I honestly think that it's physically harder to push a bike through colder air. I mean, I do wear more clothes in winter, but once I'm wearing a winter jacket and tights, it shouldn't make much of a difference if I'm wearing one thin layer or two underneath. But I bike consistently in all temperatures and it is significantly more work to ride fast as it gets colder. There was a thread about this on bikejournal once, where there was a kind of agreement that it was a combination of a lot of things: bulkier clothes, less aerodynamic posture and clothing, more energy used to keep warm, less motivation to work hard in cold air, worse road conditions, a sluggish bike with poor shifting and more friction, but also more cold wind and "thick air", all combining to make you give up and just slug along slowly. It was interesting that everybody riding in winter had noticed the same thing.