Yeah, I don't get why people do this. I don't tell those who do endurance rides of 500 miles or run 100 miles to stop, just because I might keel over if I did that...
I also chalk it up to jealousy. And yes, the comments get worse as you get older. It was acceptable for me to be at the gym morning noon and night, but as soon as my exercise moved outdoors, the comments started. Thankfully, there's a huge cycling culture around here, so I ignore my mostly unfit friends who say stuff.
2015 Trek Silque SSL
Specialized Oura
2011 Guru Praemio
Specialized Oura
2017 Specialized Ariel Sport