Are you looking directly in front of your wheel when you are going down? If so, maybe this is the reason why its so scary because you can't react to something on the road or the road itself when it comes into your field of view.

At 20MPH, I may be looking 100 feet in front. By the time, I'm going 40MPH downhill, I will be looking 300 feet or more ahead. Absolutely no reason to be looking right in front of your wheel! When you are riding on the flat, I don't think you are looking directly in front of your wheel. Chances are pretty good that you are looking way ahead 100 feet maybe 500 feet ahead.

If I were to go downhill at 40MPH and look at a spot directly in front of my wheel, I'll be a nervous wreck too.

At higher speed, you need to look further ahead and start reacting to the condition (pot holes, bumps, debris on the road) before you get to there.

Relax and try to enjoy the reward of climbing hills, that is going downhill .