I would have done the same thing. Every time I hear someone say something like this to me, about some perceived group of people, my is thinking that they probably are saying the same thing about Jews.
This is why I speak up.