I find myself assuming that small-boned short slender men are women. Body size (height, shoulders, and especially to me leg length - long legs=female) are common cues. I think sometimes working at McD's (I did in college) you just get in a grind taking orders without REALLY looking at the face you are looking at. I mostly remember the hands I put change into. In rural Idaho, there were more missing fingers than I expected (?farm accidents). But when the mistake happens over and over, it must be troubling for you. If you were wearing baggy jeans and plaid shirts - I could see it, maybe. And as if you were a man dressed as a woman, you would want to be called "sir"? - I think maybe no, but who really knows....