I've never felt weird about cycling clothes. Who wants to be sweaty and gross from long shirts or pants that aren't made for cycling?
Of course, I started riding many years after I had already been working in gyms or going there as a customer, so I was totally into lycra, and somewhat skin tight attire. I am not very modest, either. I was about 15 lbs. heavier when I started riding, but that wasn't enough to make me feel self conscious. My DH and son were serious riders when I started and it was like, this is what you wear, let's order some shorts and a jersey for you.
That said, if you are self conscious, there are several options that have been mentioned. I wear different clothes for casual rides, errands, etc. Terry skorts, Zoic knickers or baggy shorts, looser knickers. They are quite stylish and I wear some of the shorts/knickers without the cycling liner for regular casual wear.
But, you are right. No one is looking...