I do that too.
Some of these bikers are so silent that I totally cannot hear them coming and sometimes they give me a heart attack zooming right past me by all of a sudden out of the blue on a country road. Sometimes it's after a long curve and I couldn't have seen them coming in my mirror anyway. I think it's both rude and a safety hazard.