I use arm warmers below 65, without legwarmers, and maybe a wool layer instead of my summer base layer under my jersey. The leg warmers come out below 55. If it's below maybe 50, I say to heck with it and wear the jacket plus leg warmers. I've done the arm warmers under the jacket and long-sleeved jersey (and legwarmers under a pair of lined track pants) at around 35.

I find that they're useful on rides where the temperature is likely to change. (Like fall and spring, where it starts chilly but ends up in the 70s or 80s.)