I agree with what everyone has written. Don't assume the worst, assume the best. But I'll add a bit. Be prepared for the worst. Tell several people where and with whom you are going and when you expect to be back. Especially if it ends up being just the 2 of you. Make sure he knows you told several people ("We might run into so and so at that place we stop to refill water/get coffee. I told her we were coming by and she said she's working there today.") Plus bring your cell phone. That should keep you safe from major danger.

On a more likely level, if he is interested and you're not, no big deal. Be straightforward. Even thank him for his interest but have a true-ish excuse ready. But make sure you are clear. It has to be a real no, not a "oh i would but". The "but" will eventually go away and you are left with "oh I will" And have fun!! That's what riding is all about. Probably for him too.