I feel the same way. A lot of it has to do with the fact that the way I go by car isn't the way I would go by bike. Plus here there is a lot of construction that makes the feasible bike ways not as safe as I would expect (they are putting in a light rail system about 12 blocks north of me, so pretty much every road/MUP/trail system is under construction or blocked or walk-your-bike-only and I never know what to expect).

I've gotten in the habit of doing trial runs to places on weekends when I don't actually need to get anywhere in a hurry. That way if I run into a route that should be safe but isn't, I have time to find another way to get there.