I am assuming you always ride with a good helmet and that the helmet is positioned properly on your head. FOrward protecting your forehead, not ont he back of your head.

OK, here's something to practice... falling. I know sounds weird, but once you practice falling and know you won't get hurt (at least badly) hopefully your fear will subside.

Just practice on a nice soft area, lawn, sand, beach is good, you need to fall forward (as that is how most falls occur - up and over bars) tuck in your neck and shoulder, arms in never out and hopefully you will roll over onto to your side and hip without problem.

Falling does occur - it is just part of the sport - either road or mountain. Just keep riding. Remember more falls (and bad ones too) occur at slow speed then high.