What they have created for you is a hostile work environment. The fact that you have had to endure a more heated workplace since his affair, leads in a way to 3rd party sexual harassment. I'm not a lawyer.

So as everyone else has said, look for another job and leave. Quietly and professionally.

While looking for a job, you don't have to divulge to the perspective employer the sordid old employers behavior and you don't have to give them permission to call the old employer either. Though some places will do this after they hire you.

You can just say it is a small company and they are having growing pains.

I would at the very end, once you have found a job, ask boss lady for a letter of recommendation. She has no obligation to give you one, but it doesn't hurt to ask.

But leave and do it soon.