I don't know if this helps, but when I first started swimming in the pool I would be somewhat dizzy when I got out. All of a sudden I would have a hard time balancing to put my flip flops back on. This has diminished over the last few months to where I don't have it anymore.

This is just conjecture here, but I would think that the back and forth rolling motion of swimming would throw off your equilibrium, doesn't this have something to do with your ears? I could imagine it being worse in OW with more choppiness and what not. Its like a mild form of motion sickness. Just my thoughts anyway. Hope it gets better for you.