I assume you're going to shower anyway when you get home, so the issue isn't so much putting on dirty clothes over again but the possibility of bacterial growth. I'll second the suggestions to just let your shorts dry out over the day (or at least don't smoosh them into a bag while they're wet). If you try it and suddenly start getting saddle sores, then it wouldn't be very hard to go back to the 2-shorts a day system.

I don't know if this idea sounds horribly disgusting to you, but I've done multi-day tours where I've re-worn dirty shorts and haven't had any additional trouble from it other than getting chafed in the same spots from wearing the same shorts. *oversharing* If you're worried about cleanliness of the other bodily secretion sort, the key to the shorts re-wear up to several days is wearing a tampon */oversharing*