I can't add anything to the technical discussion of neck issues, but I pretty much was put in the same place by my osteoporosis. No one told me to stop mountain biking, but it was pretty much the same as you. I'm wimpy and uncoordinated (not that you are), but I love being in the woods. The one year I really improved my skills (2005-06), I spent a lot of time in the woods, but I felt guilty about losing time on the road. I got my wake up call when I went on a group mtb ride and I was humiliated in front of a lot of people, some of whom knew I was a ride leader . I realized I wasn't willing to put the time in to get better. I am also pretty risk aversive and while I liked the idea of mountain biking, I really was not the best candidate to do it.
The last time I used my mountain bike was on a dirt road in Maine's Great North Woods, 2 years ago. It was fun and not as scary, but still scarier than being on the road, to me. That made up my mind. My mountain bike was sold to someone who will really use it.