I have done that once or twice. It's usually when it's super cold and I have extra things on my head (like a balaclava and hat and maybe goggles). You know, when it's extra dangerous to ride...

I have gotten into the habit of when I hop on my bike, I pat my head. I know I look silly, but helmets are my friends