I don't have any advice for you other than - maybe with time it will get easier?? I started riding a bike less than 2 years ago (after not riding for 17 years). I remember my first time out. I was going up a hill and a guy went to pass me on the left. I went to reach for my water bottle and I swerved into him but I didn't hit him. I apologized and told him that it was my first ride in 17 years! Balance was a HUGE issue for me early on. I also remember riding with some random guy and I tried to blow my nose and I swerved into him, too! I can't remember how or when it got easy for me. It just did with time. I also marvel at cyclists who can take clothes off while riding with no hands. I am so jealous.