You will find that different people have different cold-weather riding tolerances. For ME, I am in my PI Amfib tights in the low 40's while others might not for another 10 degrees or so - and it makes a big difference to my body if it is windy/no sunshine/humid, whatever.

For me, if I ride at 40 degrees I will have on a long-sleeve wool base layer, short or LS wool jersey, thermal jacket, warm gloves. On my head is at least a skull cap under my helmet. Lower body has on my PI Amfib tights over whatever shorts I've selected - no chamois in my tights, wool socks, and shoe covers on my shoes. I don't need to take anything off - but I do have a rack and trunk bag on my bike so IF I was crazy enough to take anything off that is where it would go. I get cold very easily which is why I worded it that way I do see people riding in 40's dressed in ways that would have me freezing in sub 60 degrees...so it does differ. I like the tall Smartwool socks for cold weather riding/hiking.