I am sort of the same way, but it really depends on what field you work in. No one would accuse me of dressing super conservatively, but I do have to be aware of being "triggering," to male clients. My husband works at a software company and shorts, flip flops, etc are the uniform for the developers. DH wears nicer casual wear, including stylish jeans, etc., as he feels that he needs to be a bit above the computer geeks in terms of how he presents himself. He never, ever wears a sport coat or suit, unless it's a rare occasion he is out at a trade show or client.
In the past, I have had to discuss the fact that deep cleavage and butt cracks showing when you bend over to help a student are not OK in the classroom, especially with impressionable middle school students. Both of these conversations were with younger people, who were my assistants quite awhile ago.