I too use a front pannier on the rear (since my front panniers are smaller than the rear panniers), but I only carry one of them. OK, I know, that's a little lopsided, but I am carrying so little with me (clothes for the day, lunch & snacks, and toilet articles) that it is very little weight. And it really doesn't feel lopsided!
I've also ridden with just one (small) pannier, and I don't notice a difference either.