Follow your instinct, Snap. Comfortable clothes are one thing, but his behaviour and communication didn't really give you any clear indications. I like the idea that you chat with whoever refereed you to him and probably find another neurologist, if you can. Sending great neurologist finding butterflies your way....
~T~