I come from a long line of agoraphobics/anxiety ridden worrying Jewish mothers.... both my mother and grandmother were somewhat daredevils of their time/active at one point in their lives, and then gradually became very narrow in what they would experience, especially my grandmother. I swore a long time ago I would not be like that.
Now that I'm a therapist, I see it in perspective and when this happens on a ride, I know what it is and do the deep breathing. When I first started riding , I would be really irritable and complaining in the beginning of a ride where I felt stressed. That doesn't happen anymore, because it's just a cover up for the anxiety, and mostly because my skills are better. I don't get this in any other part of my life; in fact, I am very tough and pragmatic, and usually deal well with difficult situations at work, etc. But, it is hard to fight a mindset where you have been taught that everything around the corner is "dangerous." I've come a long way, despite my natural disability to be coordinated and a real perceptual issue. My "regular" friends think I am a wild dare devil/super athlete, which makes me laugh. But, they were brought up just like me, with no expectations of physical toughness or adventure.
I did stop mountain biking because of this, though. I like being in the woods, but my natural fear (and the fact that I have osteoporosis) made me even more cautious than normal. And I always felt guilty about not being on my road bike...