Not to put a fly in the ointment, but I did find that having the handlebar bag did make a difference in my handling. I had a Detours bag, not too big, on my Trek 5200 for at least a year and when I took it off, I realized it was affecting my stability and cornering. Of course, I am a spazz, so take it for what it's worth. I won't use another one.
I have a pannier on my Jamis, and even when it's jammed packed, once I get rolling, I don't notice that at all.