Document, document, document!

Send a letter to the race folks letting them know the objective facts (that the other groups didn't get the lecture, much less the subjective gut-feeling stuff). You can even suggest that if his behavior was based on experience of women doing that stuff,l that he should have said so to justify his patronizing attitude.

He probably won't change his attitude, but he mihgt not get that job next time around.