I have the same problem with my ears at 60 and below. I use a folded buff for 50-60 degrees, and a wind resistant fleece headband with ear flaps for 40-50 degrees, and I start start layering the buff and various headbands and skull caps below that. I've tried ear bags as well, but they were extremely uncomfortable and too rigid for me. My ear canals are the main problem.. I find that a bit of fluffed up cotton ball or loose wool [the kind sold for shoe fit/foot problems] over the middle of my ear helps also. A layer that is truly wind proof in the ear area is key for me in cold weather.