+1 on being careful of frostbite. There's a LOT of difference between 20°F and 10. Frostbite is no fun, and even a "little" frostbite on the fingers and toes can leave you with permanently impaired circulation and cold tolerance.

I just got back from 46 miles starting at 43°F and warming to 57, wool socks, full light shoe covers, and my toes were blue even after I finished my nice warm shower. The rest of my body was perfectly warm, so it wasn't my core robbing heat from my extremities. Back when I used to have to attend continuing ed programs in places that would keep their AC at 68 or 70, my fingers would be in excruciating pain by the end of the day. That's from frostbite 35 years ago. So trust me, you don't want that.

But if you want to do it and you have the gear, go for it! You might snag some of those chemical toe warmers, if as you say your toes don't stay warm in the temps you've been riding in.