Often, we are just too polite, especially around stupid men-beings.
I have found that my little bell works wonders. 'DING DING' and pass. No waiting for them to permit me to pass, just go (assuming it's safe and no one is coming the other way). They may have been clueless and figured you didn't want to pass (otherwise you would have, like they would have).
Ugh. what a drag, though.