It definitely sounds like a fit problem. I had the same trouble when I was riding a bike that was too big. If you have to stretch to reach the bars, it throws all your weight onto your arms and hands. There are a lot of things to take into consideration when fitting a bike...for example, my too-large frame was great as far as standover height was concerned (because I have long legs), but was too big because I have short arms and a short torso.

You can try swapping stems and such, but the best solution may be a different bike