Well, I think a certain amount of fear is good, but that's why I'm not a good mountain biker. I am also on the verge of "old age" and I already have osteopenia. I've had a few crashes that ended up being nothing, but a bad fall would not be good for my bones. But, on to your issue. You live somewhat near me and I can say for sure that roads are the worst I've seen them in 17 years. We went out for a 20 mile ride 2 weeks ago. There were a lot of puddles and pot holes like I have never seen. Not so much right around my house, but in the next town, which was particularly bad. I grazed one puddle which must have had a pot hole under it and my bike started shimmying and I almost went down. I was able to control it, but it was scary because we were on a main road, with cars. I was riding my hybrid, which is heavy compared to my road bike. I saw lots of people out on road bikes that day and I can't believe they were having a good time!
The advice given about the tires is good, but I would ask myself what are your limits for riding? Some days it's just not worth it. I would not ride on some of the roads around here when there was snow or ice on them. You have to decide what's important: riding no matter what or taking a bit of a cautious attitude when it's really bad out.