Umm, I think something is wrong with your pump. The internal gaskets/mechanisms/thingy-bobs should seal off the other hole when mounted on a tire valve.
It has a lever to seal it off, but when you first put it on the opposite side isn't sealed. Sorry, wasn't clear. Don't have to hold it shut when I pump