Is it a neck/shoulder flexibility problem, or a focus problem (or both)? Start with figuring out why you're doing it.
If you're weighting the arm you're turning toward, then you can do some neck stretches at home so that you can turn your neck without shifting the weight on your hands. (Consciously putting an equal amount of weight onto the other hand isn't necessarily a good idea, because then you've got a death grip on your bars. Work more on *not* weighting the hand you're turning toward.)
If it's because you're shifting your focus, work on consciously keeping your focus forward in spite of your glance back (if you practice Chi Running, they call it y'chi, which they say is what it's called in tai chi). This is part of avoiding tunnel vision and target fixation.
Either way, practice where you can do it safely - in an empty parking lot if you're confident you won't fall; on grass if you think you might.