I commute in regular clothes, but I also always wear a helmet or my 'fancy' showers pass jacket to hopefully discourge the image that I'm doing this as the result of a DUI.
my goodness! are you really worried about it? I just ride my bike and hope no one tries to run me over. (but i ALWAYS wear a helmet)