I don't wear long sleeved jerseys much either. When it is seriously cold I wear smartwool base layer and a heavier-than-a-jersey sweater from PI, and a shell.

When it's cool I wear a shell over a short sleever.