I don't ride in my drops all the time either but when I'm on a downhill I do brake from my drops. Maybe it's because of my small hands but I have more braking power from the drops than the hoods.

Since I've been using my drops on downhills I've noticed the number of riders, men and women, who do not use their drops. One friend of mine says she feels insecure but I've found that I feel more in control and secure on downhills when I'm in my drops, even on short downhills.