I wash almost everything before wearing it - just to get the smell of the sizing, inks, dyes out before. Can't hurt.
I do the same thing since I'm allergic to sizing and many dyes, if I try to wear something without washing it first I end up with a really icky itchy rash.