I also wear a screaming yellow vest with reflective stripes on it, and I do look like a dork in it! Especially with the purple helmet, the gloves and the red plastic crate on the back of the bike
. With very few exceptions, drivers give me plenty of clearance when passing. I haven't had any real problems with rude drivers in general, the worst I've had is comments/catcalls from guys on the sidewalk (and those are sometimes funny). The only time I don't wear the vest is if it's a bright sunny day and I happen to be wearing a really bright top such as my shocking pink t-shirt.
2011 Surly LHT
1995 Trek 830