So all your cycling friends are only those that race?
It's doubtful that even if you left racing at any point in life, all those racing cycling friends would disappear if some are good regular friends, unless racing is the only focus for them.

I dunno. Don't worry about what others think. Do what you love to do.

Tell us a little more about yourself about cycling, other than just the weight issue.

And do you cycle just for commuting, travel, too? Doing other types of cycling will get you exposed to other cyclists who are strong cyclists but they don't race..and may not necessarily worry about weight for racing purposes. the world of cycling accommodates all kinds of people.

Wishing you inner peace and more enjoyable cycling, pressure-free.