You probably do not have it pulled up enough. Work the suit up from the bottom and make sure it is all the way up under your armpits.

And just for the record, I always swim with a wetsuit if allowed, yep a 1/4 mile swim is short, but you are still faster.