I've never been asked that question. I have been asked "Do I have to wear Spandex?" a whole lot, though.

I do agree that it can be a barrier to riding. It's expensive. It looks funny. You can't just go ride your bike. It sets expectations that you need a whole bunch of special gear to ride a bike.

As to why we get the comments, I think it's two-fold. It's a very visible mark of being "other," those brightly-colored jerseys and Lycra shorts. It's not mainstream like yoga or even martial arts. The other thing is that most people cannot imagine wanting to dress in such form-fitting clothes. I talked to so many people who said "I read that bike shorts help, but I'm not comfortable wearing something that tight."