I vote for quality of life over the ideal job. One idea is to relocate so you can have both. That is what we did 15 years ago with no regrets.

penny