It's not all in your head and it would drive me nuts too. I'd feel like I was being pressured to hurry up even if that's not the intention. Cassandra has a good compromise and he can wait at a predetermined location if he needs to make sure you're okay. Or, he can back track and then you can lead.