You just need to work on it, not stop riding. I have the same issue, in that I found I look down, as Oak describes. I already knew I had horrible visual/spatial skills, so working on looking up is just something I do to improve it all. And, I second what Oak said about peripheral vision; I am always scanning to the sides when I ride, especially now that I commute through my town center at the end of rush hour. I constantly check my side and rearview mirrors when driving, too, and use only my mirrors for changing lanes. I have had friends yell at me (sigh) for not turning my head and looking when I change lanes on the freeway. My experience is that here in Massachusetts people never use their mirrors, and either whip their heads around to change lanes, or worse, don't look at all and just go. Although this is slightly off topic, I think it speaks to the fact that a lot of people don't use their peripheral vision.
2015 Trek Silque SSL
Specialized Oura
2011 Guru Praemio
Specialized Oura
2017 Specialized Ariel Sport