I agree, feeling comfortable with the doctor is important. From what you told about the meeting, the comment of not knowing what he was looking at would make me uncomfortable, too, especially in combination with the general vibe you got from his office.