If it's cold and rainy, something that's light but wind/water repellent I've noticed does the trick. That way you don't feel like you're freezing the whole time but you're not burning up either.

For when it's pretty cold, I like the Pearl Izumi super lightweight jacket they had out a few years ago....I can't seem to find it now....