My brakes all get a good solid workout! You are not alone. Confidence (miles) in my bike helps, as does repetition of the same hill (you know what's coming and begin to get more comfortable). I'll still never be a 50mph downhill person, and that's ok!