I commute with a backpack every day, but it's very small and most important, it's very narrow, so it sits nicely between my shoulders. It will not take a full change of clothes, but maybe it would take summer clothing minus shoes. (Maybe you can keep shoes at work?) I usually just carry lunch, keys and wallet, but I end up doing some grocery shopping on the way home and having a bulging pack for the final half mile :-) And yes, it gets hot.

Conversely I hate carrying anything but a very light load on my back, but I'm not too happy about panniers either. I'm fussy about bike handling :-) Panniers are probably great if you need the space, but you could try to get away with a small pack and judicious packing first. I will typically keep several changes of clothes at work and bring home one garment at a time to wash. They need washing very seldom since I shower as soon as I get there, and have a quiet office job.