I have a one-a-day 4 or 5 shot soy latte (unsweetened soy w/no sugar or sugar substitute added) and I am almost positive this is not the reason I am 15 lbs overweight. I am also positive that if I gave it up I would not be any fun to be with at all (and it's debatable now. ).

Skim milk is a good source of calcium. The no sugar syrup is probably not the healthiest thing in the world - maybe try weaning yourself off that. But I doubt that it is having a big effect on your weight loss efforts.