Jolt, you might ask your LBS how low you can go with those tires. Fifty psi still sounds underinflated to me if the max is 80. You won't blow them if you inflate them higher than 50 psi; I promise. While I agree that you don't need to inflate a tire to max pressure, especially if you're light, I'm afraid that you still might be at risk for a pinch flat. And whatever you do, always inflate your tires before every ride. You'd be surprised how much air they lose just by sitting there.
As for speed, since it sounds like you're beginner, all you really need to concentrate on right now is riding more. Don't worry that much about how fast you're going. Greater speed will come naturally as you get in better shape, learn to choose the best gear for the conditions, and increase your cadence (rotations of the pedal per minute) up. Have fun with it.
Live with intention. Walk to the edge. Listen hard. Practice wellness. Play with abandon. Laugh. Choose with no regret. Continue to learn. Appreciate your friends. Do what you love. Live as if this is all there is.
--Mary Anne Radmacher