He's looking back at you because he doesn't want to let a big distance get between you. I think it's sweet! I would worry if he never bothered looking to see where you are. Remember, we all see things differently. But he did clarify his friendly intentions, so now the "annoyance" feeling is coming just from your side.