A lightweight Terry bolero would probably work for you but it would take up more room in your jersey pocket after you shed it.

TE used to sell some Castelli arm warmers that were lightweight. The Sheila Moons might be a little more substantial than you want (I have a pair from a few seasons ago -- I like them but they are warmer than the Castellis).