I would wear separate knee or leg warmers for temperature swings like that, and pull them off once it starts getting warm. But if your knees don't complain about the temporary cold there's no need to.
I have the same kind of thing here, figuring out what to wear in to work in the morning that won't give me a full backpack going back on a sunny afternoon. I usually need to pack short gloves and a thin headband for the ride home, legs and arms do well with leg warmers and arm warmers or thin jacket.
Running is different again, lower speed, less wind and higher intensity means I generate more heat and can wear less.