I suspect it is only a temporary fear since you've been away from cycling for so long, and, of course, as adults we KNOW what can happen. That puts things in a different perspective.

Ride as fast as you feel safe and things will improve as you gain more confidence on the bike. Fear isn't a bad thing and as SmittyKitty pointed out does serve a purpose. I've only been riding a couple of years (just turned 54) and going downhill was terrifying to me when I started. It still does if the pavement isn't in decent shape OR if it is a very steep downhill. Focus on relaxing, keep your shoulders out of your ears (if you do that), and keep looking down the road and what comes next so you can anticipate shifting. This will, hopefully, prevent you from focusing too much on your speed. Don't use your speed to guide your braking but the terrain, conditions, and of course, your gut.

This helped me, others will be able to add to it.