I feel the same way. I use my Jamis Coda for errands and going to appointments, meeting people for lunch. I do miss commuting, and since I am not working now, these trips are my "commute."
Just an unscientific poll. Last week, I rode the Jamis to an appointment in West Concord. It was pretty busy going through the 2 "downtown" areas on my trip. I was wearing a skort and a non-cycling wicking sleeveless top. No problems. On Monday, I rode to the gym on my road bike, going through the same area, although a longer ride. I had my Camelbak on, used as a backpack, but I was dressed like a roadie. I had 2 angry driver incidents (swearing, yelling at me for taking the lane), in an area where there are tons of cyclists and it is not unusual to see commuters.
I think people are less angry at cyclists who look like "regular" people.