I am somewhat risk averse, NWG, although I sure do a lot more than 10 years ago. Your plan sounds very sane to me, and you will still be riding. Ultimately, risk was the reason I quit mountain biking.
It was 53 degrees at 5:20 AM, when I went out for a little ride. Just a little ride over to the other side of town, but I did not do the longer loop, as I have to be at work at 9, so only 10 miles. I was wearing arm and knee warmers, a vest, and my lightest weight long fingered gloves. My head was actually cold in the beginning, which is a slight downhill, but it was fine in the end.
I love this kind of weather. The humidity is gone and it should be this way until Sunday, when rain and more heat/humidity will arrive for next week. Bah.