Do you own a pair of sleeves. I know Peal Izumi and other companies make sleeves that you just pull up over your arms to just underneath the short sleeves of your jersey. When you get hot, you just pull them off! They also make the same thing for leggings as well to where you pull them up to just underneath your shorts.They're really convenient.
Lately, it's been around those temps and a little cooler and all I wear are the sleeves and maybe a lightweight jacket over top, so if I do get hot, I just pull it off. As for the legs, I wear shorts, since the legs are constantly moving. If you start when its sunny out but finish your ride around dusk or later, wear or bring something lightweight that keeps you warm. Also, when do you ride and is that during the high temps? If not, again carry something light so you won't get the chills. That's my advice, hope it helps