I'm the same way. Whenever I travel new places by bike I'm always a bit nervous/anxious. I think it's just- like you said- are "in the environment" and more vulnerable.
It's gotten better over the years and I've found that I'm less nervous/anxious than I was when I first started riding for transportation. I think the more you do it, the better it gets. And it's a great confidence builder for me when I make a trip/do an errand by bike that I wasn't sure I could.

How did your trip go?