I've always been taught that the downhill rider yields to the uphill rider for exactly the reason that you describe. Unfortunately, around here, not everyone seems familiar with this convention :/