I'm in the same boat...
I work for a large law firm, and am burning out fast. Billable hours rule my life, and I have no time to do what I care about (bike, hike, etc). I am seriously considering several positions which will literally be a 50% pay cut (or more). But...I'll do work I believe in (I don't always sleep well right now), I'll have a very short commute (could probably bike) and I'll have time to do what I love. I don't have the answers yet. I had hoped to hang in there until my loans are paid off, but I'm not too far away on that front. And I know I can't do my best work where I am too much longer (there are major changes underway which mean more $$, but less free time and much more stress).
So no, I don't think you're crazy (unless I am too). Just make sure the time is right for you and that if you don't love it, you have options.
Good luck!
Most days in life don't stand out, But life's about those days that will...