I think it's okay to tell him you'd rather run alone... especially since you don't run with anyone else, either. Do you think he would take that badly? He must know other people who don't like to run with others.

I much prefer running alone. I can't pay attention to all the little signals from my body if I have to pay attention to someone else.