What a great question! As a bigger lady (I'm about 5'10" and 210 pounds), I don't have to tell you how cruel people are to anyone with a little fat on their bones (and this is a HUGE part of the population....sheesh). But I can tell you for CERTAIN that a couple of those people, although they were thin were NOT fit. Like, not at ALL. But of course, thin doesn't mean fit. I am just talking about their uppity-ness, looking down their noses at me because I was bigger and the unspoken taunts of "wouldn't you LOVE to look like me" sort of thing.
I feel fit. I feel GREAT. I am not thin, never, ever will be, and don't ever want to be. So the average person is never going to look at me and say WOW, she's FIT!! But my boyfriend knows for certain that I AM (hehe, yeah, he knows!) and I know I am.
Being "fit" is a personal thing, I think. It's how you feel...it's what you can do on the bike, running, whatever it is you do.