If I've been riding at all long or hard, my hair is already sticking up through the vents in the helmet. So from that POV it doesn't matter if I take it off or leave it on.

For me it has more to do with where I'm going to put the thing. For a quick errand or pee break I'll leave it on my head but maybe release the buckle. (If you ride in a forward lean or tuck, your helmet strap should be a little uncomfortable when you're upright! It's not much use if it's just going to slide to the side of your head in a crash.) But if I don't have to think about where to put it, then that's 15 seconds that I didn't add on to a 30 second errand.

If I'm going to be inside for a longer time, like in a restaurant, and there's secure outdoor parking where it won't get jostled and possibly dropped, I'll hook it over my handlebars. Or in the saddlebag of the commuter bike.