maybe you have a short torso? I do not have that problem with any of my shorts and i am 5'3 1/2" .
I wear some smalls and some mediums.