I think you have to balance the water temp vs the length of the swim. If the swim is less than 1/2 mile I'll go without the suit even if the water is pretty cold. For anything longer, I want the bouyancy and warmth of the suit.

That being said, can you rent a suit and try it out? You may find that you really like it and it's worth getting one. Or you may find that it's not. (There may be suits available to rent at the tri.)