I think the thing you most need to be concerned with damage-wise would be numbness. Otherwise, if it doesn't hurt, you're fine. There are women on this forum who ride lots of miles sans chamois, and have for years.