I, too have had problems recently with cysts/ingrown hairs. I make sure I take my shorts off and shower as soon as possible after my ride. I use benzoyl peroxide cream (like for pimples) and that really works well. If I can get anything out of the cyst/ingrown hair, I will try and then put the cream on 3x a day and this really seems to make the cyst go down fast. I also try not to wear underwear unless I have to (difficult for me!!) and keep the area as dry as possible.