I use a backpack too. I actually like it better than loading down the back of the bike.
All I carry is a change of clothes and a lunch and my wallet and cellphone it's just not THAT heavy.