I have a little vial of pepper spray without a holder. I use one of those short 3" loop bungy cords with a ball on one end. I loop it around my stem and tuck the spray in one strand of the elastic, holding the spray vertically snug against my stem. Very handy to grab and it stays out of the way.

Don't forget to bring the spray in with you if you go into a store, bathroom, etc. You do NOT want some kid fooling with it on your bike, spraying themselves, getting hurt, and possibly suing you as well. Do not leave it unattended on your bike in public places.