If you are wearing a bike jersey with pockets you can stuff it in there as well.

Another option is to get arm warmers.


These are some from here on TE. You can wear them with any short sleeve top. When you warm up you can just push them down as you ride and leave them around your wrists if you have no place to store them.

I ride a lot with them on my wrists because I warm up on the climbs, get cold on the descents.

V.