My partner rides with a back pack if he has too take clothes to where he is going... he usually gets a sore back.
I very rarely ride with anything on my back, cause it takes so much pleasure from the ride.

When I start commuting regularly (which will be when I am no longer a school taxi service) I am going to get panniers on a steel frame bike set up in the same configuration as my road race bikes.

I guess we both ride in racing position, so that may affect the discomfort levels, but I also feel unbalanced with a load on my back...