I see your point on this, Absolut, but I admit that I'm like Owlice, I appreciate a warning and having some type-A rider buzz me p*sses me off (some of my responses to this sort of thing are, "passing on your right," "please warn next time," or "thanks for the warning").
When I ride down at Haines Pt., I try to pick times when I'm pretty sure the racer/hammer types won't be riding there (earlier in the morning, for example, on a weekend). If they pass me without warning but give me plenty of room, I don't care that much.
Ride defensively. I've had these aggressive riders buzz me on uncrowded MUTs, for Chrissake. It's stupid and pointless. Who cares if you can pass me, just keep a safe distance. There are a lot of people who apparently, were raised by wolves, and lack common courtesy. The worrisome thing is that this is a safety issue.
.