Unless you live on this stuff I wouldn't worry :-) My understanding is that it takes a pretty radical diet to harm a baby, and even then it's more likely that the mother suffers and that the baby is just fine. Alcohol, nicotine, sure, stay away. Caffeine, reduce. My experience was quite simply that I didn't want that much caffeine, so my regular two cups a day suddenly went down to a small half cup once I was pregnant.

Best of luck :-)