I really think you're OK. I did 5:30 AM spin class at least once a week for years and my shorts were fine in my bag all day.
Yup, Indy, I have become more like you. I admit I like clothes, but most of my purchases are for athletic type stuff. It's one thing if you just do one sport, but I have collected quite a collection. I used to x country ski in my cycling tops and base layers, but my yearly forays into running have provided me with pocketless thermal tops for winter sports, too. I am a sucker for cool athletic clothes. Finding them in my size is a quest, but thanks to Athleta, I have some nice things for sports besides cycling.
My regular wardrobe has become more multi functional and mostly black!