Doesn't matter whether they are faster or not, it's a shared use trail and everyone should be riding for the conditions (which includes other users) and their own skills. If you are riding and can't move over to let them pass, just yell that you'll move as as soon as possible. If you are walking, get out of their way as soon as possible. Yield to uphill traffic but if someone is in the middle of a technical rocky downhill, prudence may dictate that you stop. It's a judgment call for everyone's safety.

Under no conditions should you try to get off a trail if it is not safe (like a cliff edge).

Relax and just tell them you'll do what you can when you can. If they're speeding around a blind corner and out of control -- their problem not yours.

One day, some unknown guy might even say "thanks, sweetheart" Made me laugh all day.