I definitely go downhill slower than most of the people in my group. We have some pretty steep hills and I don't see why it is necessary to go so fast that I feel unsafe. The only time I really rip down a hill is on rollers when I see a big hill that I am going to have to climb, and I want to get as far up as possible. Besides, I know the hill is going to slow me down.

I have noticed that as I continue to improve my riding, I am comfortable going downhill a little faster than I used to. So, just gradually increase the downhill speed in increments when it feels ok. Don't try to make it a mental block or force yourself to do something that would be unsafe for you. Everyone's comfort level is different. It would be way worse to hurt yourself and be unable to ride. Anything under 20 mph feels fine now. Anything over 28 mph is downright scary (for me). I catch up in the flats and uphill.