Ugh ... my husband used to do the same thing. I confronted him, enough times that it stopped being an issue. At this point I honestly can't say whether he doesn't do it any more, or whether I just quit feeling guilty. Just go and ride.