I really think it has to do with hair, in our (US) society women are "supposed" to have long hair. I've had little kids ask me if I was a boy or a girl when I was at the pool wearing a swimsuit.