I LOVE to descend! It's my favorite and I'm really good at it. I do get a little freaked out sometimes at what could happen if I crashed, but I just try to ride as safely as possible and enjoy myself. I just set my Garmin to tell me my fastest speed. When I did Heartbreak Hundred, I got up to 47 at one point. Yikes! Pretty sure that was my fastest. Had no idea I had gotten that fast and never thought I'd feel comfortable doing that speed on anything other than the tandem with dh. Yet, I recall feeling fine and enjoying every moment of it (after the stupid climbing I had just done and knowing there was a lot more where that came from).

Stay in your own comfort zone, whatever's comfortable for you. If you feel uneasy, don't push yourself just to keep up with others or "get over your fear". Keep your speed down and eventually you may begin to feel more at ease. Read up on descending skills. That may give you some confidence.