I left a stressful job in banking to a job at a non-profit. My pay (and benefits ) went down. I certainly have moments of regret about the "losses," but I am much more relaxed... and I have time to do fun things like ride my bike and my horses! So...

If I may... I think your tag-line says it all...

It's only worth it if you're having fun.