I bought a thin flat purse with a top zipper that fits well in my panniers. It solves the issue of losing small stuff to the depths and I don't have to collect all that stuff when I'm away from my bike.

I nabbed a set of Racktime TravelIt panniers on Ebay for ~$50 last year, and they've been fantastic. Good size, plenty of pockets, integrated rain cover, and the Ortlieb rack attachment system.