I'm so unfashionable that it's not even funny, so take this with a grain of salt.

I made a small drawstring bag that slips easily inside my pannier with all my purse junk in it. Most days, this is sufficient since I'm only leaving the building for site visits (informal, get dirty sort of affairs). I've considered leaving a more fashionable purse at the office, just in case I need to go off-site for a meeting or whatever.