So I'm not cycling in ordinary street clothes/chic

[Susan]

I don't think that the guy wanted to offend you. Of course it's none of his business what you want to wear on your bike.

I think it's maybe just a slightly different point of view if you use your bike for everyday transportation, all errands, trips to the city and so on. I don't like to go to a cafè or a restaurant in my cycling clothes. If I was to wear cycling clothes every time I use a bike I would never wear any normal clothes at all.
While I use cycling clothes and a helmet when doing cycling as a sport, I don't use any cycling-specific gear when I use my bike for transportation or driving into the city. I think a lot of people here do so (and I think it has nothing to do with being a "fashionista").
Still I don't care if anybody wants to wear something else.

I too notice that some people seem to be frightened or offended by more "professional" cycling gear, I get far more negative comments when in my cycling clothes. A lot of people seem to think that wearing cycling gear means that you go fast and act as if you were about to run them over even when you go really slow or even stop as they pass you - strange.