I can't imagine what I would say if I had been you!!
I, too, think time is way too precious to "kill". I wish I had time to ride during the week these days but the workday is way too long. I am already logging miles on the trainer during the week and hoping for a DRY weekend in order to ride on the road.
You did a wonderful job of ignoring his comment and focusing on the positive. Keep pedaling and don't sweat the folks who have no clue.
"When I'm on my bike I forget about things like age. I just have fun." Kathy Sessler
2006 Independent Fabrication Custom Ti Crown Jewel (Road, though she has been known to go just about anywhere)/Specialized Jett