the hardest part about wearing just a sports bra and wanting to put something on after the swim is getting dressed while wet. If you have a vest that is easy to pull on over a wet body and zips up- that might be just perfect. Jump in the shower with your bra on, get out, and try to put that vest on immediately and see if you have any trouble. If not- that might be a great solution.
I have seen so many people struggling in transition to pull on shirts over their wet bodies- and it never goes smoothly.
As for running in a sports bra- you could just wear the vest, but you're right- there will be all shapes and sizes out there and nobody is going to be judging your outfit...or your skin/body.