I've never felt uncomfortable in cycling shorts in public, but I have been in places where I thought it might be more appropriate to be a little more covered up, mostly in Europe.
I have noticed a phenomena where it's mostly women who stare. I chalk it up to wishing they could do what I do, jealousy, etc. I notice this more as I get older, where other women my age stare a lot. It happened at the B and B we were at this past weekend, in the Berkshires. On the second morning, we went to breakfast in our bike clothes, because we were on a time schedule. We got stares like crazy. I chalk this up to the fact that most of these people were from that large city about 200 miles south of here, and they aren't used to seeing hordes of lycra clad riders of all ages, shapes, and sizes, all of the time, like we do.