Well, as far as the comment about men, I am always surprised by some of the guys in my cycling group. They have bellies and are very strong riders. Maybe because they are older and have been riding for years? Not sure, here. This group does like to eat, though...
I have had experiences similar to you, GLC. I look thin, but not skinny. I have some "untoned" fat around my hips and outer thighs that will just never be gone, without more work than I am willing to do. I look good in clothes, but not in a bathing suit! Anyway, once I went to my gym for a body fat assessment. The guy thought his calipers and the electrical impedance thingy were broken when he kept coming up with 15.5% for me. This was years ago, too, when I weighed even less. Even now, it rarely goes above 18%. I guess this is the effect of almost 30 years of exercising!
Some people just will never get over the stereotype of skinny-minny is good. My (very unfit and slightly overweight) friend was bragging that she bought some "skinny" jeans and that they "had them in her size." When I told her that i couldn't wear them, because they won't go over my cycling calves and thighs, she thought it was terrible and didn't understand how someone could be thin and not be able to wear these jeans. I told her I would rather look like an athlete and that muscle is a good thing.