A year ago I started biking. I was very scared of downhills. I went down them all s-l-o-w-l-y, braking all the way. I felt I should not go any faster than I felt safe going.
As the months went by, I rode faster, a little at a time. I just naturally gained confidence and gained bike handling skills.
Now I can zoom down most big hills in good control....hills that used to terrify me last year! I laugh at some of the "huge hills" I used to brake down all the way....they look like little teeny bumps now!

My point?- don't worry, downhill speed will come naturally as you get more skills on your bike. It will come all by itself. Don't go any faster than you feel safe at.

Remember, riding downhill while controlling your speed, braking, and balancing your weight is a valuable bike skill too!