I loooooove pits so much, there were always way too many at the shelter I volunteered at, and they were all sweet hearts (though usually untrained and terrible at walking on a leash). Small terriers, on the other hand, still make me curl my lip and take a step back.
ah, but my preferences are irrelevant.

If you feel uncomfortable with the dog, that is *very* valid. Do not let yourself be bullied or brushed off by your co-workers. Understand the owner of the pit may take your concerns personally, so try and make it about you more than the dog. As in, "I feel very uncomfortable. I'm sure your dog is very sweet, but I have no idea why he growled at me and he scares me". Any dog can be dangerous, try not to make it about his breed. If anything, people might write you off as an irrational pit bull hater.

Dogs can growl over any little thing, maybe someone was in there harassing him before you. Maybe he was guarding a toy. Maybe he was doing an impression of Arnold Schwarzenegger. Who knows. But you are totally right to be bothered by it, because he also could have been telling you f-off, and that isn't a dog's job.

I hope you manage to work something out.