Yes, wash the gloves.

Last summer I put my winter gloves away for the summer w/o washing, and almost couldn't make it through the first ride with them come fall. They reeked!

'Course, I use them for wiping sweat, etc. off my face, so I shouldn't have been surprised...

So far, washing hasn't done my gloves any visible harm. I line dry them, for what that's worth.