If it's windy or cool at the start of a ride, I wear a vest over a short sleeve wool jersey and arm warmers. The vest keeps your core warm, but can fit easily into a jersey pocket if the day warms up, or even just be mostly unzipped.
Arm warmers are more versatile than a full sleeve jersey for the same reason - easy to get rid of if you don't need them.
Veronica