IMHO: People at the gym who have time to worry about what you're wearing are NOT busy enough exercising.

Of course, that's not to say that I haven't on occasion decided this or that outfit over-accentuated some flaw that I didn't want to go around flaunting at the gym. But still.