If you're going faster than someone else, then pass.

It does drive me crazy when someone passes me, then pulls right in front of me and rides slower than I was going. So then I have to pass them. Inevitably I come to an intersection where I have to stop, and just as traffic clears or the light turns they come up and pass me and sit in front of me again so I have to pass them again. This drives me crazy. Usually at that point I pull over and wait for a few minutes because I want them far away from me.

But if it's clear that you're riding at a faster pace, then go ahead an pass.