I have a pair of the REI wind/waterproof pants. They are designed for bikies so the ankles are more fitted and the knees have a dart in them so there's room for the pants to move over the knees when pedalling. I wore those all last winter with leg warmers/shorts and was fine.

(ok...I'm a whiney baby when it come to the cold but I wasn't going to give up riding and those worked for me!)