I've used one pannier on the left side (to be wider to cars) for about 35 years. I honestly cannot tell the difference between one pannier on the left vs 2 panniers when I'm riding.

No damage to my bikes, either.