I don't size down either, but do find that it pays to try stuff on. Even within a given brand, the sizing might vary from style to style. I like my baselayers to fit close to my body, but not to the point the I feel constricted.