You should be cold when you start out - cause you will warm up. So, don't wear the jacket at first, just a jersey and armwarmers for instance, ride the hard part where you get sweaty and then put your jacket on for the downhill. Sounds like it is just not cold enough for the jacket right now for the entire ride.
spoke