I agree that bike fit has something to do with it. But, so does conditioning and experience...

When I first started riding, I had major balance issues. Drinking from the water bottle only happened at intersections and there was no way I could even wipe my upper lip without veering half off the road.

The first improvement I saw was when I got a new bike that fit me. My first one was way too big and I felt very unstable. Secondly, my second bike was also better suited to my riding style as it was less 'twitchy' because it wasn't as aggressive. It rode a bit more stable.

Then as I worked on core strength, I also built up stability on the bike.

Lastly, time and experience has helped. It's been 2 years (next week, actually!) and I can drink, adjust things on my bike/person, swat at bugs, zip and unzip my jackets, turn to look behind me, etc...all without swerving. I still cannot go 'no hands', but since I don't plan on winning any races, I don't really need too!

So I would suggest 1) check bike fit 2) improve core strength and 3) give it time.