I think the point about the type of bike you ride and how you are dressed is a good one. I have never experienced overtly rude behavior from a driver when I'm on my clunky pink bike, dressed for work, although people still do stupid things. Instead I get a lot of waves and good mornings. When I'm on my road bike I only get that from other cyclists.

(Drivers are even friendlier when I don't wear a helmet, but I decided that friendliness was not a good tradeoff for safety.)