The use of "my" grates much more than the use of "girl". Some people use "girl" all the time. I consider it a diminutive or casual term, fine for friends or in a very casual sense, but not really appropriate in a lot of settings.
Yes. I belong to NO one. For a man to use it to refer to a coworker smacks of women-as-property, and I'm just not tolerating that.

Girl, gal, I don't care, until you put the word "my" in front of it.

Karen