By ManFriend, do you mean your current Significant Other?
If so, I find the comment really sad, and that's because I'm in a good mood, or I'd find it insulting.
My own SO is much, much more of an athlete than I am and will ever be (he's also more obsessive!), but he never fails to express his pride in my own achievement (like learning to swim by myself in three months and doing my first tri etc.). I'm not sure I still have the balls to claim "athlete" for myself (never considered myself a sports gal and still have identity issues surrounding that!) but he sure would argue against me if he knew that's what I think.
My core training coach keeps telling us that, in her view, we're "elite" athletes even if we're not actually "elite". (She does train pros though.) To her, there's so, so few people in the population that can do even a fraction of what we do, we should be proud of it. (Like: standing on all four on two exercise balls... how smart do you look when you do that?!?!)