That is the dumbest thing I've heard... the stupidity of people never ceases to amaze me.
I once had a daycare worker at my gym ask me if my kids had different fathers, because at that time they looked quite different, one blonde, one very dark. I replied, "Gee, no, I would have had to work pretty fast for that, since they are only 2.5 years apart."
As far as the weight comments, just keep riding. You are probably a better rider than a lot of people. I've been dropped by a lot of riders, who didn't fit the stereotypical mold of what some people think a good rider looks like.