Well, yes... that much extra material (if it fits as loose as a jersey) will slow you down.
And I know for me... it would be downright annoying. Being a competitive swimmer in H.S., and focusing so much on getting rid of drag... it's even a challenge for me to deal with the shorts and top I have now.
Keep in mind... when it gets wet... it will get BIGGER too.
Now, the issue that you face is that you have a bust... and all those darn tri tops are made for these really skinny women (you could be thin, not saying your not!) with no boobs... not really skinny women with boobs.
I would wonder if you could have it taken in at the waist by a tailor. Might be a bit of a stretch though.
Maybe you can go to a cycling shop and see how other sizes/brands fit? I know it was a struggle for me to find one that fit good in the waist and the bust (thin on top with 34C's).
The top I have now is a little loose, but I just ignore it as best I can.
"Life is not a journey to the grave with the intention of arriving safely in a pretty and well preserved body, but rather, to skid in broadside thoroughly used-up, totally worn out, and loudly proclaiming: WOW WHAT A RIDE!!!!"