I commute with one heavy pannier all the time.
Though it's recommended that the pannier be on the left side, between the bike and traffic to create a visual barrier, that didn't work for me. I put my right foot down first when stopping, and having the weight on the left side of the bike made me very unstable. So I put it on the right and all is fine.
I never even notice the lop-sidedness.
2007 Rivendell Glorius/Trico gel with cutout (not made any more apparently)
2005 Specialized Sequoia Comp/Specialized Dolce
2006 Kona Cinder Cone/another Trico gel
1986? Bridgestone mixte/Brooks B72
1991 Bridgestone 300 Xtracycle/Terry Gelissimo