Catrin, you will find that people will generally continue with the "you are nuts" talk even as you prove to them that you are continuing to ride. My friends that I've had for years just shake their heads. It wasn't "nuts" when i went to the gym all of the time, but cycling is! I think in their eyes going to the gym is what "women do," but cycling is just too dangerous. One of them continually tells me what she "used" to do, like hiking, etc., but now she doesn't even want to sweat a little. I once went for a 5 mile walk with her and she came with a bath towel draped around her shoulders, to wipe off the sweat.
Keep riding.