People can have a weird way of projecting their own insecurities onto you. I don't know if it makes them feel better to do so, but it really is their problem, not yours.
I've gotten comments about being too skinny my whole life. Even during a very stressful time of my life when I had trouble keeping weight on, in spite of a healthy appetite, I got dangerously thin.
We all have our burdens. I've never found it productive to compare who's got it more difficult.
2014 Bobbin Bramble / Brooks B67
2008 Rodriguez Rainier Mirage / Terry Butterfly Tri Gel
2007 Dahon Speed Pro TT / Biologic Velvet