+1 on avoiding any further encounters with the scary man. It sucks that you have to change your route, but the potential for this to escalate into something ugly is simply not worth it.

One other thing to add to the excellent advice that others have posted -- if you haven't already done so, it might help you feel more prepared to deal with a potential conflict if you were to take a self-defense course, especially one that addresses how women are approached/attacked. Ask the police officers when you talk to them, if any of them teaches self-defense, or if there is a community college near you that offers these courses.

I took an excellent course when I was in grad school a few years ago -- they talked about avoiding potential conflict/danger; asserting yourself as a means of warning someone who might be bothering you that you are not "easy prey"; breaking different kinds of holds; basic punching and that it's good to practice not broadcasting your punches. And we got plenty of scenario/role playing practice at all of this throughout the course.

Hopefully you never have to "use" any of this stuff; but in my experience, it's helped me to stay calm in a crisis and know that if someone came after me I would know how to handle it.