I always wear normal cotton underwear under my shorts/onepiece for rowing. I think that most of the men's crew go commando though. I know they did at university because they hid our shoes while we were training one day. The next day they came back from a training session to find their underwear flying from the boathouse flagpole (yes, childish but satisfying).

As for swimming in your rowing shorts - mine are always fine.