Yeah, this seems to be a common human trait. The good news is that you can use it to your benefit at other times. If you're ever in a skid in your car or otherwise feel like you're losing control, look at where you want to go. It actually helps you get there.

Is the arrow pointing to the hole from the side or from the front? If it's from the side, I'd suggest that you fixate on the middle of the arrow, but if it's from the front, well...

Maybe you should bring a can of spray paint the next time you go and draw yourself a nice dashed line (or a happy face) in the middle of your narrow-but-safe path. That way you can fixate on that and that'll guide you through safely. Or at least, that would be the theory.