+1 on panniers. You want your center of gravity as low as possible. 20# on your back is more than enough to affect your handling.

I've ridden a short, flat couple of miles a few times with a 3 gallon water jug (~25#) in a backpack, and it was tough to get started from a stop (175 mm crankarms on my commuter didn't help that a bit). I definitely wouldn't want to do it in traffic.