I know from experience that loose shorts are a pain - and even dangerous- when you take off and the waist gets caught on the front of your saddle!

Where does your waist measurement fall on the sizing chart? If it is borderline too, it might be that this brand just doesn't work for you.