Part of riding balance is the distribution of your weight between the saddle, handlebars, and (not so much) pedals, the three places where you contact the bike. If you have too much weight on your hands you may not feel comfortable letting go of the bar at all. Beyond that, the geometry of the bike will help determine how well it tracks when the bars are unweighted as also will the tightness of the headset.

I can ride with either hand and grab my water bottle or stuff from my pockets, I can lean forward far or sit up almost straight, barely touching the bars, and all is well. As soon as I remove both hands from the bars the bike feels out of control. I used to be able to ride no-handed as a kid, so I think it has something to do with this bike.