It sounds extremely challenging. I've had only a taste of some of those types of dirt roads (always with lots of rocks) in western MA, and they always involve a steep climb.
There will always be someone fitter is a truism. It's hard to let that one go, even for me, who is not fit compared to you.
The mental aspect of racing is probably the hardest thing, from an outsider's perspective.