I commute with one heavy pannier all the time.

Though it's recommended that the pannier be on the left side, between the bike and traffic to create a visual barrier, that didn't work for me. I put my right foot down first when stopping, and having the weight on the left side of the bike made me very unstable. So I put it on the right and all is fine.

I never even notice the lop-sidedness.