Downhills are the reward for all the hard stuff!
I learned to love downhills by riding the same one every day in our local park. Once you know where all the bumps and potential trouble spots are, you can really fly. Now I play around with my aerosdynamic position and see if by changing it (ie moving elbows in) I can get faster. It's nothing crazy - about a 4% grade for half a mile.
I had a bad time at first on the downhill in traffic on the way home from the park. Sometimes my hands cramped from pulling on the break. Repetition is the one thing that helped me stop worrying the most.
My coach says it's safer not to break. He also told me to pedal slowly to add stability to the bike going downhill. You can drastically lower your speed on a downhill (and avoid the brakes) just by sitting up and going back to the hood from the drops.
You will learn to love the downhills:)