A vest?

[maillotpois]

I do what Veronica describes on practically EVERY ride (except for the wool jersey part - I do synthetic jerseys).

Arm warmers/vest/SS jersey work for every type of weather, unless it's raining (wear full showers pass jacket from the start), or going to rain (bring thinner rain jacket along in a pocket), or ridiculously cold (wear full thermal jacket and maybe even rain jacket on top). I have several weights of vest (thick wind and cold stopper, thinner for cool, not cold days) and pair with varying thicknesses of arm warmers.

That said, I don't know that I'd use a primaloft type vest for cycling. It just seems too thick and not breath-y enough.