People who are lactose intolerant can sometimes handle goat's milk because it has much less lactose than cow's.

People with a true allergy, it will depend on the protein that's the trigger - actually I'm much more sensitive to goat's milk than cow's, but much less sensitive to sheep's milk cheese.