I don't buy a size down; if they are meant as a base layer, they should be sized to fit snugly.

As with all clothing, there are variations. A Smartwool small is perfect for me, while an Icebreaker small is too tight.