Perhaps you could benefit from taking a clinic or some private lessons--something that could help with both your skills and confidence.
I will admit that I'm a total newb. For a long time, I've been relunctant to try MTBing for some of the reasons you mentioned. I finally decided to try it and immediately signed up for a clinic that's next weekend. I've also made DH promise that he won't push me to ride trails beyond my skill level. I'm sure some do just fine pushing themselves in that way, but I know myself well enough to know that it would likely just spook me at best and get me hurt at worst. Perhaps you need to just dial it down a notch or two, ride and your skill level and lay a more secure foundation before going out again on more challenging trails.