I seem to have the same problem. I bought some Champion sports bras that have a little extra padding up front, and that seems to help. It's not a temp. thing with me, either. I think my body is just so excited to be exercising, that it can't contain itself. My body says, "Yippeee- you've taken me out for a run, bike, swim, tennis, walk, game of miniature golf," etc. It's the girls' way of thanking me, I suppose. I joke, but I, too, get embarrassed by it sometimes.
Let them stare. Who cares? It just means they're looking at a fit, powerful woman doing something they only wish they could do.