To help prevent whatever infections cause saddle sores, I've heard folks recommend drying them with the chamois exposed to direct sunlight. That will probably take care of all the nasties.

Maybe tuck as much lycra UNDER the chamois to prevent excess UV damage while it's drying outside.

Edit: In the winter, better come up with another solution. A frozen chamois doesn't sound too comfy!