your body weight distribution does have an effect. Additional things are

head tube angle, the amount of rake in your fork, the wheel base length. A recreational bicycles are set up to be more stable, softer to ride/more comfortable. They are not meant to make sharp fast turns. Bikes made for crits have shorter wheelbase and smaller rake and steeper head tube angle (I think not 100% sure). The overall effect of crit bike tends to be "squirrly".

Old Binachi was squirrly I could only ride maybe 20 feet before I had to grab the handle.
Old Panasonic DS-6000 (model name?) was really stable and i could ride for really long distances if I wanted.
Litespeed Tachyon was a relaxed tri bike. also ride for really long distances.
Orbea Dama Race '05 a little more squirrly than Litespeed or the Panasonic.
Guru Chrono'alu a full blown serious tri bike for TT use (I just tried don't own) squrirrly.

So it may not be you ability or lack thereof, it may be that you have a racing bike instead of recreational cruiser.

smilingcat