I do unhook and carry my panniers around. Yea, a bit of a pain but I've done it for so many years now.

We tend to make several mini trips to different food shopping places during the week. Grocery shopping is seldom in 1 singular bike trip. Since each of us are on our bikes often during the week, we each take the initiative while rolling along homeward, to buy some food.

In retrospect, now I realize that we don't even draw up grocery shopping lists in advance..since it's shopping for just 2. So our cycling grocery shopping trips and efforts very rarely duplicate too much of same food or buying the "wrong" food.